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CHAPTER I j 

I 

INTRODUCTION AND HISTORY OF THE PROJECT S 

— 



1.0 Introduction 



This report describes the development of the International Project 
for the Evaluation of Educational Achievement (IEA) in recent years. In 
particular, it describes in detail the work undertaken between November, 
1966 and February, 1969. This period has been the first stage of the 
second major phase of IEA’s work. During this stage, international 
groups have constructed and pretested cognitive and affective instruments 
to evaluate the outcomes of learning in six subject areas at three dif- 
ferent points in the school systems in twenty countries. The first point 
in the school system comprises the 10 year old level, the second is the 
14 year old level, and the third is the final year of secondary school or 
the pre-university level. In addition to the evaluation instruments, 
questionnaires have been constructed to include measures of variables, 
descriptive of the student and his home background, the teachers in school, 
the curriculum and instruction, and the school characteristics. Procedures 
for the data collection and data processing have been worked out for the 
field work in the period 1969-71. 

Volume I of this report includes a description of various parts of 
this phase of the study accompanied by supporting documents which are to 
be found in the appendices. Volume II of the report is the collection of 
instruments and administrative manuals which have been produced in this 
phase of the IEA work. 

1.1 Initiation of IEA 



In the middle fifties, groups of educators and educational re- 
searchers from different countries met at the UNESCO Institute for 
Education, in Hamburg, Germany, to examine problems such as those con- 
cerned with school structures and organization, selection processes, 
examinations and failure in school. Two important publications emerging 
from some of these meetings were edited by Hotyat (1962) and Wall (1962). 
Throughout these meetings there was a growing awareness of the need to 
establish evaluation techniques which would be valid cross-nationally . 

At the same time, more or less independently of each other, several re- 
searchers in the United States (Anderson, Bloom, and Foshay) began to 
consider the possibilities of undertaking such research. 
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1.2 Pilot Study 

In 1958, researchers from several countries came together at 
a meeting in Eltham, England, chaired by Dr. W.D. Wall of the National 
Foundation for Educational Research in England and Wales, and also at 
the UNESCO Institute for Education in Hamburg. At these meetings it 
was decided to carry out a pilot study to discover if an international 
research project would be administratively possible and if the results 
could be meaningful. Research Centers from Belgium, England, Finland, 
France, Germany, Israel, Poland, Scotland, Sweden, Switzerland, the 
United States and Yugoslavia took part, the target population was all 
children in these countries aged 13:0 - 13:11, since this was the last 
point where all of an age group were still in school in all these countries 
In most cases children in schools or areas which were known to be near to 
the national mean and standard deviation were tested, and thus there was 
no strict probability sampling. All in all, 9,918 pupils spread over 
eight languages were administered tests of reading comprehension, 
mathematics, science, geography and nonverbal ability. Altogether 
there were 120 test items, and the tests were edited by culling from 
existing tests, for the most part, items thought to be appropriate. 

The results of this pilot study were reported in a book edited by 
Foshay (1962) . 

Despite the evident limitations of the pilot study in terms 
of sampling and test construction, some of the findings were very 
interesting in the sense that they were unexpected. It was found that 
when the variance between countries in a subject was expressed as a 
percent of the average within-country variance , then that for reading 
comprehension was 6 percent; mathematics, 15 percent, science, 5 percent; 
geography, 15 percent; and nonverbal ability, 12 percent. This was 
surprising, since it was expected by some that the international variance 
for nonverbal ability would be least. 

Another interesting finding was that boys varied more from 
country to country than girls. Professor R.L. Thorndike drew up national 
profiles or patterns of achievement. A country’s pattern of achievement 
showed that country’s achievement on specific tests and subtests relative 
to its own overall level of achievement. The profiles indicated, for 
example, that French pupils performed relatively better in mathematics than 
in science, but that the reverse was true in Germany and the United States. 
Despite the shorter length of schooling of pupils in Scandinavia (pupils 
there begin school at the age of 7), the 13 year olds there performed, in 
general, as well on the tests as children in other countries with longer 
schooling. 

Apart from supplying information which was extremely useful in 
the generation of hypotheses for the IEA project, the answer to the main 
problem of the pilot study was that it was possible and meaningful to 
carry out an international project of this kind on a fairly large scale. 
Work then began immediately on the first main phase of the IEA project. 
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1.3 Mathematics Study (Phase X) 



In 1961, the heads of research organizations and university 
departments of education, in twelve countries, met together to form the 
Council of the International Project for the Evaluation of Educational 
Achievement (IEA) . These countries were: Australia, Belgium, England, 

Finland, France, Germany, Israel, Japan, Netherlands, Scotland, Sweden 
and the United States . 

The Council decided that the first phase of the project should 
consist of an examination of attainment in one subject area only - 
mathematics - together with the collection of data concerning the fac- 
tors that are associated with performance. 

From the outset it was agreed that the project should be a 
co-operative enterprise. Matters of policy have been determined by full 
meetings of the Council. Interim decisions have been taken by an elect- 
ed Standing Committee, and the day-to-day administration has been the 
responsibility of the Chairman, the Technical Director and a full-time 
Co-ordinator. 

The participating research center from each country made it- 
self responsible for the administration of tests and the collection 
of data within its boundaries. In other respects resources were pooled 
in the sense that wherever particular facilities existed they were placed 
at the disposal of the Council. For example, the United States repre- 
sentatives were able to arrange for all the data processing to be car- 
ried out at the University of Chicago Computation Center; England supplied 
an expert in sampling to advise on all the sampling procedures; and 

the United States, Belgium, and Sweden supplied mathematicians to serve 
as consultants to the test constructors. The costs of the operation 
occurring within each country (e.g. in carrying out the testing program, 
etc.) were borne by the country concerned, and the international costs 
were defrayed by the United States Office of Education. 

The major purpose of the inquiry was the measure achievement 
in mathematics, and to relate this to those factors in home, school and 
society which might have been expected to be associated with it. In 
determining these factors, the investigation had, of course, to lean 
heavily upon the findings of previous research carried out in the 
main within particular educational systems. The project represents an 
attempt to assess the "efficiency" or "productivity" of different 
educational systems and practices. The countries participating were 
the twelve .mentioned above. Two major points in the school systems were 
sampled: 
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a) 13 year old (both age and grade populations) , since this 
was the last point in all of the systems where one hundred 
percent of an age group were still in full-time schooling, 
and 

b) the pre-university grade students. 

In some countries students at a major terminal point intermediate 
to 13 year olds and pre-university students were also tested. Apart from 
the international mathematics tests being administered, an opinion booklet 
(description scales of the school and class learning environment as well 
as various attitude scales,) student questionnaires, mathematics teachers 
questionaires, and national case study questionnaires were also administered. 
All in all, just over 133,000 students from 5,400 schools were tested and 
questionnaires were completed by 13,500 teachers and 5,400 school principals. 
The number of single pieces of data collected was just over 50 million. 

It was possible to examine the "effectiveness" of different 
educational practices in school and class organization, curricula, 
performance. Furthermore, outcomes were related to social, economic and 
ecological factors. The hypotheses tested can be grouped under the 
following four main headings: 

1. The relation of school organization to mathematics achievement. 

2. Curriculum and instruction. 

3. Support for education. 

4. Social, economic and ecological factors. 



The results have been published in Husen, (et.al. ,1967) . 

Apart from many findings relevant to the improvement of mathematical 
education in each of the participating countries, it should be noted 
that it has been possible to construct a "fair and valid international 
nathematics test and through it to establish national norms for sever a 
target populations on the international test. The implications of this 
for comparison of pre-university grade students seeking co ege en ranee 
in countries other than their own are obvious. 

A Data Bank (Wolf, 1967) has been established and is open to all 
bona fide scholars. 
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| Most of the national research centers have prepared reports on the 

| Mathematics Study in which they have highlighted the implications of 

this research for their own schools. By the end of 1968 national reports 
| had been produced by the following research institutions: Australia, 

| (Keeves, 1968); Belgium (De Landsheere, 1967); England (Pidgeon, 1967); 

f Finland (Kuusinen, 1967, and Lyttkainen & Pavliinen, 1969); Japan (N.X*E.R. , 

[ 1967), and Sweden (Hultin, 1968). 



r 

I 



I 

| 




£ 






t 



1 . 4 A Six Subject Area Study (Phase II - Stage 1) 

Already in 1964 it was clear that it would be necessary to repeat 
the Mathematics Study in other subject areas. Therefore IEA asked various 
of the existing National Centers to carry out for them what were called 
"Feasibility Studies" in certain subject areas. National Centers set 
up committees who furnished IEA with reports on the possibilities and 
worthwhileness of conducting surveys in selected subject areas. It was 
evident that Science was a strong candidate for future study as a subject 
which would not be too different from mathematics. Therefore no feasibility 
study was carried out in this area. However, the Swedish National Center 
produced a report on Civic Education (see Appendix A-l) , the Finnish 
National Center one on English as a Foreign Language (see Appendix A-2) , 
the Belgian National Center one on Mother Tongue (see Appendix A-3) , 
and a special U.S. Committee headed by Professors Foshay and Purves 
produced one on Literature (see Appendix A-4) . Another U.S. Committee, 
which was headed by Professor John B. Carroll, then of Harvard University, 
supplied a report on French as a Foreign Language (see Appendix A-5) . 



These reports were considered by the IEA Council in February 1965, 
and it was decided that an attempt should be made to construct instruments 
in each of the subject areas. There were some doubts expressed by some 
members of the Council as to whether valid international instruments could 
be constructed, in some areas, especially in Literature and Civic Education. 

Not only was it decided to attempt to repeat the Mathematics Pro- 
ject in these six other subject areas, but also to extend considerably the 
background information collected (the "inputs" into the school systems) 
to try to account for more of the variance of the "outcomes" of the school 
systems (in terms of the students’ performance on the tests) and by so doing 
to lay the factual basis for decision making for school reform - both nation 
al (within nations) and international (comparability of "standards" etc. 
between nations) . 

Phase II was split into two stages initially; Stage 1 was for the 
construction of the instruments, and Stage 2 for the field work, the ana- 
lyses, and the write-ups. 
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Stage 1 began in November 1966 and continued until March 1969. 

Although an attempt was made to construct instruments in all six subject areas, 
it was anticipated that IEA might not be successful in each one of these 
areas. It was agreed that the instruments produced in Stage 1 would be 
reviewed at the end of 1968 in order to decide on those subject areas which 
would continue into Stage 2 for the field testing. It was found necessary in 
fact to split Stage 2 into Stage 2 and Stage 3, as the burden of testing 
all six subject areas at the same time was excessive. In Stage 2, therefore, 
only Science, Reading Comprehension ane Literature will be field tested, and 
Stage 3 will consist of English as a Foreign Language, French as a Foreign 
Language and possibly Civic Education. 

The main aims of Stage 1 have been the construction of the follow- 
ing instruments: 

- Tests in each of the subject areas for 10 year olds, 14 year 
olds, and pre-university students, as well as for major ter- 
minal populations between 14+ and the pre-university students; 

- Accompanying student, teacher, school and national question- 
naires ; 

- Descriptive and attitude scales. 

At the same time, sampling designs and proposals have been developed, 
hypotheses formulated and data analysis programs planned. 

The nations participating in Stage 1 have been Australia, Belgium, 
Chile, England, Federal Republic of Germany, Finland, France, India, Iran, 
Italy, Japan, Netherlands, Poland, Scotland, Sweden, Thailand, United States, 
New Zealand . 
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1 . 5 Formation of International and National Committees 



I One* that IEA had decided to attempt to construct cognitive 

1 achievement tests in the six subject areas, the first step was to 

form international subject area committees, the members of which 
would be drawn from various countries and be knowledgeable in the 
field of test construction in a particular subject area. To this 
i end all National Centers were circularized to propose at least one 

person for the subject areas in which they were participating and 
to send full information about that person. Once the information 
was received, it was reviewed by the Bureau, and in the first in- 
stance chairmen were selected, and then a selection made from other 
| nominees such that different cultures and different experiences 

| were represented within one committee. Once the names had been 

1 agreed with the chairmen, letters of invitation were sent to pro- 

posed members. Where a member declined the invitation because of 
| too many work commitments, further discussion took place with the 

I chairman of the committee and in some instances other nominations 

I which had been made were taken into consideration and in other cases 

| new names were suggested. The composition of committees was final- 

ised by October 1966. In some cases members were not able to con- 
| tinue with their work, in which case other persons were put on the 

| committee, and hence in the committees as they now stand, there are 

| occasionally more than one representative of a nationality on one 

| committee. In some cases as a member could not attend one meeting, 

the opportunity was taken to bring in another expert in the field 
| for just that committee meeting. Pages 19-29 of Booklet 6 in Volume II 

| present the names and biographical notes of the members of the 

c International Committees . 



f 

l 

| National Centers were requested to form national committees 

in each of the subject areas in which they were participating. 
Again members were expected to have some knowledge of test con- 
| struction, but at the same time the committee should be so com- 

i posed as to represent the different types of schools within any 

one designated population within the country. Where National Cen- 
ters had a person from their country on the International subject 
area committee, they were asked that this person should be chair- 
| man of the national subject area committee, or at least a member 

I of it, since this person was to act as a link between the two 

committees. 

I 

I 

| 
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How National Centers organized the national committees and 
the number of members on the committees was at their own discretion, 
and from page 91 -99 of Booklet 6 it will be seen that in some 
countries there were many members, and in others only a few. In 
Booklet 6 the names and positions of all members of national subject 
area committees are presented. 



As will be seen from the separate subject area reports below, 
there was a continuous exchange of ideas between the international 
and national subject area committees. The international committee 
decided on the rationale of the test construction and asked for 
comments on this from the national subject area committees. Con- 
tent analyses were required from the national subject area com- 
mittees, and this indeed was an enormous amount of work. Further- 
more, national committees were requested to submit test items to 
the international committees. Once the pre-tests were in first 
draft form, the national committees were asked to comment on them 
and the pre-tests were revised in the light of these comments be- 
fore being submitted for pre-testing in the countries. The national 
committees were also asked to comment on the pre-test item analysis 
from their countries to the international committee. The inter- 
national committee proposed a final test version and national 
committees were again asked to comment on these items and the final 
versions were revised in the light of their comments. In many 
cases it was members of the national committees who were responsible 
for the translation of the items into their own language for testing 
purposes. How active national committees were depended to a great 
extent on the National Center, but all committees worked hard and 
much of the good quality of the final items is due to their dedica- 
tion and professional interest. Table 1 shows the National Centers 
participating in the different subject areas in Stage 1. 
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As the data came back from National Centers , so the Interna- 
tional Committees met again and prepared for the next round of data 
collection/pre-testing. After pre-testing they met again for the 
finalization of the instruments. Committees met on an average of 
two or three times a year. Between meetings, however, committee mem- 
bers carried out a great deal of work on their own. 

At the same time work was undertaken continually by the IEA 
Central Staff together with some consultants on the construction of 
questionnaires, attitudinal and descriptive scales. (See Chapter 8), 

In order to provide a more detailed and more robust concep- 
tual framework for the IEA and similar studies, two meetings were 
held under the title of "The Cross-National Conference on Education, 
Manpower and the Economy" (U.S. Office of Education Grant No. 0E-1- 
061967-1989). In these meetings outstanding scholars from the various 
social sciences were brought together to review the type of research 
being undertaken, and at the same time to suggest from their own dis- 
cipline point of view the types of hypotheses which could be decided 
in such surveys, together with the types of variables which had been 
missing from the Mathematics Study, but which could be included in 
further studies. These were fed to those people responsible for the 
questionnaire construction, (See Chapter 8)* 

The Central Staff and the Consultants also worked out the whole 
procedures for the administration of the field testing, and these have 
been written up in Manuals 1, 2 and 3 (See Volume II, Booklet !)• 



I At various junctures throughout Stage 1, Bulletins were issued, 

: which brought together the main technical papers and policy decisions 

taken over a period of months of most importance to National Centers, 

; such that all of these were within one document. From the beginning 

| of 1968 newsletters were sent out by the Chairman every quarter. Two 

■ examples of these are given in Appendices B— 1, B-2. 

| 

I The content analysis in most subject areas was finished by 

I the end of 1967, and the pre-tests produced by Spring 1968. The pre- 

I testing took place between February and October 1968 and the item analyses 

| were returned between June and December 1968. Final versions of the 

I instruments were ready between December 1968 and March 1969. 
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1.6 IEA Organization 

[; 

In order to be able to receive money and to become a legal 
\ organization in its own right, the Council of IEA decided that the 

organization should become incorporated. It was decided to become 
f incorporated in Belgium, following advice from the UNESCO Legal 

| Department. IEA thus became formally incorporated on June 13, 1967, 

j and the Statutes are attached. (See Appendix C) . The structure of 

; IEA is given in detail in these Statutes. The Council has met 

during Stage 1 approximately once a year, as has also the Standing 
| Committee. The Bureau, consisting of Professor Torsten Husen, 

| Professor B.S. Bloom, and Mr. D.A. Pidgeon, has met approximately 

| every two to three months to take the major decisions required 

[ between Council meetings. 

I 

| Dr. Postlethwaite has been responsible for the day-to-day 

| administration and for heading up the permanent staff. On April 

| 1st, 1968, Dr. B.H. Choppin of Cornell University was employed as 

| full-time data processor, and on January 1st, 1969, Mr. J.K. Hall, 

I Head of the Surveys Department of the National Foundation for 

f Educational Research in England and Wales also joined the IEA 

jj full-time staff to work in data processing. 
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1 . 7 Participation in IEA 

Booklet 6 of Vol. II is the IEA brochure which gives full details 
of all persons involved both internationally and nationally in IEA. 

Canada, Hungary, Israel, the Phillipines , and New Zealand have 
also applied to IEA. The U.S.S.R. was formally accepted into IEA in 
December 1967, but has not found it possible actually to participate in 
any of the work so far and negotiations are still continuing with other 
countries . 

In December 1968 National Centers committed themselves definitely 
to Stage 2, Science, Reading Comprehension and' Literature as given in 
Table 2 as well as tentatively to English as a Foreign Language, French 
as a Foreign Language and Civic Education for Stage 3. 
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CHAPTER 2 
SCIENCE 



2 . 0 Introduction 

Several factors influenced the pattern of the work in Science* 

(a) Science is now almost universally recognised as an essential 
component in the education of pupils in secondary schools 

and, at an appropriate level, is being increasingly introduced 
into the curricula of primary schools. 

(b) In most countries the content and method of science educa- 
tion are undergoing critical examination and appraisal; 

syllabuses are being changed and new emphases are being given 
to the methods of study. In brief, one can say that school 
science is being given a more contemporary outlook, and this 
means that much new, and often untried, material is being in- 
troduced and much traditional material discarded, also that a 
spirit of scientific enquiry is being infused into the work, 
in which methods of working and the attitudes developed are now 
regarded as at least as important as the knowledge acquired. 

(c) The traditional differences between countries, and between 
schools within countries, are widening because of the 

vastly different stages in reform that have been reached. Thus 
in some countries, where the curriculum is in the hands of the 
individual school or the Local School Board, some schools will 
have had several years of experience of the new materials and 
methods while others will still be following the traditional 
science courses. 

(d) Probably the most significant difference between the science 
education of one country and another lies in the nature and 

extent of the practical work undertaken in the laboratory. At 
one extreme laboratory experiments are regarded as the essential 
experience upon which the theoretical structure of the subject 
is erected. At the other, practical work is seldom, or never, 
attempted and when it is, its purpose is to illustrate or rein- 
force what has already been learned. 

It is clear that science education is in a state of flux, healthy 
as regards the pupils being educated but imposing serious difficulties 
from the point of view of an objective, comparative study. Obviously 
the first thing to be done was to obtain, as far as was possible in the 
circumstances, a synoptic view of the science curricula of the partici- 
pating countries, both present and what was likely in the immediate 
future. 
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2 . 1 First_Stage 



Accordingly a form (Appendix D-l) setting out a list of subject 
areas and teaching objectives was devised for completion by the 
National Centers for each target population. In this way it was hoped 
to obtain sufficient areas of agreement between countries on which to 
base cognitive tests measuring the attainment of specific teaching ob- 
jectives. 



■ 0 

me 



Several points need to be made in connection with this form, 
(a) ^S.aumbe^^nature^and^method^of^exjression^of the subject 



areas. 



An important decision was taken at the very beginning, viz. v 
that with the exception of the so-called science specialists of 
Population IV S, all pupils would take a test in Science com- 
prising unidentified sections of earth sciences, biology, chem- 
istry and physics. In other words, science education was to be 
seen as a whole, even though this meant that some pupils would 
be unable to answer certain items in their papers. 



The form in which the subject areas were set out implied a 
certain approach to the teaching which might not have been equally 
appropriate to every country and certainly differed from one 
branch of science to another. Similarly the listed subject areas 
were not comparable in extent or importance. Thus, whereas in 
physics the subject divides itself fairly naturally under such 
headings as heat, mechanics and electricity, the arrangement of 
the biological subject areas under three levels of organisation, 
the cell, the individual and the population, is of a different 
kind. Nevertheless, the list of subject areas was agreed upon 
and subsequently functioned satisfactorily. 



(b) S*®_£eaching_objectiyes 



Two totally different approaches were possible. On the one 
hand a list of objectives as they might be set out by a practis- 
ing teacher concerned only with his actual job of teaching, and, 
on the other, a list arranged as a hierarchical structure which 
would facilitate subsequent analysis of test items. The first 
alternative was adopted, but comments from National Centers made 
it quite clear that in the long term this was a mistake, although 
from the point of view of gaining an impression of the science 
education in the different countries the objectives as set out 
were not without merit. 
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(c) ^e_£re8ent_^d_future_gositionfl 

National Centers were asked to assess the importance attached 
to each topic on a three point scale and to attempt to identify 
the teaching objectives related to it. It was recognised that in 
many cases the replies would be largely impressionistic and sub- 
jective but at this point it was essential to cdllect as much in- 
formation as possible. 

The task of compiling the subject area/teaching objective grid was 
a formidable one undertaken at its first meeting by the International 
Committee for Science; the grid was amplified by two tables (Appendix D-2 
Table I and Table II ) which expanded the list of objectives and the 
list of subject areas as might be appropriate for the three target popu- 
lations respectively. 

At the same time a preliminary list of the areas of interest and 
the hypotheses to be tested was drawn up and circulated to National Cen- 
ters so that they could gain from the beginning an overall view of how 
the science side of the project was being developed. They were set out 
as follows: 

I 1. To what extent are the newer science teaching methods based 

upon investigation and pupil enquiry achieving better results 
than traditional methods? 

2. Are any differences observed more marked in achieving some 
objectives than others? 

3. To what extent is the success of science teaching, and par- 
ticularly the newer methods of science teaching, dependent 
upon 

a. the material conditions under which science is taught, 
e.g. , provision of laboratories and apparatus, 

b. the time allocated to science and the fraction of it 
spent in practical work in the school timetable, 

c. the provision of laboratory assistance, 

d. the training teachers have received and the facilities 
for refresher courses? 

4. Is a more adventurous attitude towards future careers in 
science, as indicated by a wider range of desired careers, 
favored by the newer teaching methods? 



II 



1, What differences can be detected between the sexes in their 
attitudes to, and their achievements in: 

a. science as a whole, 

b. the physical sciences and the life sciences. 

2. To what extent are any of these differences related to: 

a. social variables , 

b. home background, including conditions and interests, 

c. school administration and organization, 

d. teacher variables? 



Ill 1. To what extent is achievement in, and attitudes towards, 
science related to: 



a. the level and rate of advance of technological develop- 
ment in the countries concerned, 

b. rural and urban environments? 



IV To what extent is achievement in, and attitudes towards, science 
related to school organization variables such as comprehensive 
and selective systems of education? 

2.2 Second St age - Topic and Objective Specifications 

Ten National Centers completed the grids by the specified dates 
and in addition they, and one or two that were not able to complete 
the grids in time, contributed valuable comments on both subject areas 
and teaching objectives. As a result, a shortened list of subject areas 
reasonably common to all those countries from whom information was 
available was drawn up for each population and sent to each National 
Center (see Appendix D-3) with an urgent request that test items cover- 
ing the specified common subject areas and judged to be suitable for 
the population specified in the country of origin should be submitted 
without delay. 



At this point the weakness in the expressed objectives from the 
point of view of test construction was felt acutely and National Centers 
were given some guidance on procedure and asked particularly to produce 
items measuring the higher scientific abilities and those testing special 
abilities such as the design of experiments or the handling of scien- 
tific apparatus. 



About 2000 items from twelve different countries were received 
in time for the second meeting of the Science Committee when the first 
draft tests were to be compiled. 
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Host of the items received were concerned, as requested, with the 
common subject areas and therefore potentially usable, but they were 
very uneven in quality and a high proportion were not in a suitably ob- 
jective form in which they could be marked mechanically. More serious 
still, however, were the facts that far too few items measuring higher 
abilities were received and that there were very large discrepancies, 
both in quantity and quality, between those contributed by different 
countries. It was difficult, therefore, to achieve the desired balance 
in the construction of the draft tests and to give a fair representa- 
tion to every country. There was, however, the safeguard that all the 
items were to be pre-tested in each country and that National Centers 
would have ample opportunity for further comment. 

Modification^of^the^scheme^of^teaching objectives 

As indicated above, the response from National Centers and the 
need to categorise test items had suggested that it was essential to 
devise a scheme of objectives arranged in some hierarchical pattern 
and adapted to the particular nature of science education. Alternative 
schemes had been submitted by two National Centers and these were con- 
sidered in relation to the well-known scheme given in "Taxonomy of Edu- 
cational Objectives. " As a result a scheme of four hierarchically ar- 
ranged objectives was adopted. These were stated briefly as A. Func- 
tional Information, B. Comprehension, C. Application, D. Higher Pro- 
cesses (see Appendix D- 3 for amplification). 

Two points about this scheme deserve comment in addition to that 
given in the appendix. Category A, Functional Information, was intended 
to eliminate items of pure recall as yielding little of value to the 
project, and, at the same time, to make room for certain practical 
abilities needed for the acquisition of scientific information, such 
as observation and the interpretation of observations. Category D f 
Higher Processes, was introduced to avoid the difficulty that, in 
science, the categories Analysis, Synthesis and Evaluation of the 
Taxonomy cannot be arranged as a simple hierarchical system. 

It is intended that the categorisation of test items selected for 
inclusion in the final tests should be validated in a separate opera- 
tion carried out by specially briefed teachers in each country. In the 
meantime the tests were built up on the basis of objective categories 
agreed by the Science Committee. 

2.3 Third_Stage - Pre-testing 

Select ±on_of^test_±t©ms_^or_comraerit_and_eveiitual gre-tes ting 

At a meeting of the Science Committee held in Hamburg in Septem- 
ber 1967 a first selection from the items submitted by National 
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Centers was made. First the items were selected from the point of view 
of appropriate subject area coverage, a pre-determined balance of teach- 
ing objectives and, as far as possible, equal representation of the 
contributing countries. The final criterion applied was that the item 
was, in the view of the Committee, potentially a good one. The items 
were then put into a reasonably common form (multiple choice with five 
alternative responses) and new items devised to fill the most obvious 
gaps. It must be stated, however, that item construction is a skilled 
process and that because of the limited amount of time available to 
them the Committee wa6 forced to accept some items which had obvious 

deficiencies. 

Following this meeting, very rough drafts of the pre-test forms 
were sent to National Centers for comment. In compiling these forms 
the balance between the items assessing the four behavioral categories 
was aimed roughly in the proportions set out below, but in fact this 
balance was not achieved. 



Population I 
Population II 
Population IV and IVS 



A3 

A3 

A1 



B2 

B3 

B1 



Cl 

C2 

Cl 



D1 

D1 

D1 






In spite of a very tight time schedule, helpful comments on the 
pre-test drafts were received from almost every one of the participating 
countries and, on the strength of these, a small group of three members 
of the Science Committee prepared the final pre-test forms. The items 
were now much improved in quality, their presentation was more uniform 
and, in general, they gave the impression that real progress was being 
made towards a valid, final testing instrument. 



The following table summarises the materials sent out for pre- 
testings 



Number of Items 



Popu- 



No. 

of 

Sub- 



Behavioral 

Category 



Major Subject Area 



i.aw.on Tests Total 


A 


B 


C 


D 


Earth 

Science 


I 4 


104 


38 


35 


18 


13 


13 


II 8 


192 


54 


66 


35 


37 


1 


IV 8 


126 


37 


22 


36 


31 


19 


IVS Bio. 4 


96 


20 


28 


18 


30 




IVS Chem 4 


92 


19 


41 


19 


13 












21 


20 





Biology 



Chem- 

istry 



Physics 



40 

65 

46 

96 



( 51 ) 
64 62 

30 31 



92 



92 
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Pre-testing of science items was carried out by sixteen countries 
early in 1968 on the basis of the scheme set out below: 





Population I Population II 


Population IV 


Australia, Belgium, 
Germany, India 


Sub-tests 12 3 4 


12 3 4 


12 3 4 


Chile, England, 
Iran, Netherlands 


Sub-tests 12 3 4 


^567 


1 * 5 6 7 


Finland, Italy, 
Japan, Scotland 


Sub-tests 1234 


12 7 8 


12 7 8 


Poland, Sweden, 
Thailand, U.S.A. 


Sub-tests 1234 


3 5 6 8 


3 3 6 8 



For the specialist tests in Science (4 Biology Sub-tests Bl t 2, 
3 and 4; 4 Chemistry Sub- tests Cl, 2,3 and 4; 4 Physics Sub- tests PI, 
2, 3 and 4), a slightly different rotation was required. 

SPECIALIST TESTS 



Australia, Belgium, Germany 


B1 


Cl 


PI 


B2 


C2 


P2 


India, Chile, England 


B1 


Cl 


PI 


B3 


C3 


P3 


Iran, Netherlands, Finland 


B1 


Cl 


PI 


B4 


C4 


P4 


Italy, Japan 


B2 


C2 


P2 


B3 


C3 


P3 


Scotland, Poland 


B2 


C2 


P2 


B4 


C4 


P4 


Sweden, Thailand, U.S.A. 


B3 


C3 


P3 


B4 


C4 


P4 



Before undertaking their pre-testing programs National Centers were 
given advice on how to deal with difficulties of translation, the ques- 
tion of the use of popular and scientific units and of unfamiliar plants, 
animals and materials. In general, National Centers were given a rea- 
sonably free hand to substitute, at their discretion, suitable alterna- 
tives within the structure of the item , on condition that the Co-ordin- 
ator was informed and that copies of the translated items, as set, were 
made available to the Science Committee for their v.se in drawing up the 
final tests. Singularly few difficulties occurred; those that did were 
fairly easily overcome and most of them arose from the correct alterna- 
tives in the question being given away in the process of translation. 

It might be amusing to speculate on the reasons for the absence of dif- 
ficulties due to purely local conditions. Why should the science taught 
be much the same all over the world? 



2-7 










% 



l 



y 






% 




National Centers also received guidance on the administration of 
these cognitive tests so that certain standard procedures were followed 
and the same kind of statistical data were produced. 



Pre-testing£ the_item_anal£®:L® 



The results of the pre-testing , carried out on judgment samples 
ox 100-200 pupils for each population and sub-test, were analysed in- 
dependently by National Centers and submitted to IEA Headquarters in 
Hamburg on a prescribed form* In addition, the results 

for each item from different countries were collated on a single form* 



The Science Committee had thus available to them something in the 
region of 700 items, for each of which a difficulty value, and a dis- 
crimination index for each country that had been able to carry out the 
pre-testing to time were known. The range of difficulty for each item 
between countries was also known. The number of countries which was 
able to submit their analysed data varied from one sub- test to another, 
but in most cases it amounted to at least six and the combined results 
can be regarded as providing a satisfactory basis for the final selec- 
tion of the test items. 



Selection and final editing of the pre-t eg tedi terns 



There were roughly four times as many pre-tested items as were 
needed although these were unevenly distributed between the four beha- 
vioral categories and some had been shown to be unsuitable on statis- 
tical grounds. A fair degree of selection was therefore possible and 
there would still be a reserve of items that could be used on some fu- 
ture occasion. 



The final selection of items and their arrangement in tests was 
carried out by the Science Committee at a meeting in Hamburg Ist-lOth 
July 1968. Their aim was to select items in such a way as to achieve 
a spread of difficulty which meant that each country was fairly repre- 
sented and to use only those items which discriminated well. At the 
same time the subject areas and the behavioral categories had to be 
suitably represented. 



For each population, therefore, a frame was constructed in which 
the rows represented subject areas and the columns the behavioral cate- 
gories or teaching objectives. The aim was to have at least six items 
in each sub-test that could be drawn from the frame for the purposes of 
separate analysis. For Populations I, II and IV the subject areas cho- 
sen for this purpose were earth sciences, biology, chemistry and physics, 
although within this very broad classification adequate representation 
was given to practically all the topics listed as being reasonably com- 
mon to all countries as a result of the original analysis of the cur- 
ricula grids. For Population IVS sub-division of the separate sciences 
was of course possible. 
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Where other factors were equal, items were selected which were 
judged to giro information about the scientific climate in which a stu- 
dent was being educated in preference to those that measured the effect 
of specific instruction. 

2 . 4 22!“2EH2ii22-2£-SJ2_£i52l-i22j2 

Several other considerations had to be borne in mind in the com- 
pilation of the final drafts. 

Anchor _^tems 

The first was the need to include anchor, or bridge, items which 

were common to two populations and which would therefore allow 
comparisons between populations in point of time to be made. In the 
final tests there are 11 such items between Populations I and II and 
11 between Populations II and IV. 

j2^^gS^g^~$ gg ^g_g^_ggTected Population IVS> items 

In order to meet the difficulty of defining a science specialist 
in the pre-university years, for whom the tests for Population IVS have 
been devised, and to enable comparisons to be made with corresponding 
students in countries where such students are not easily identified and 
inhere the Population IVS tests will not be taken, some items common to 
Populations IV and I VS are desirable. So that an adequate field of 
subject matter can be tested within the time allocated and without af- 
fecting unduly the nature of the tests for Population IV, a system of 
rotating sub-tests has been introduced. Six sub-tests each of two 
biology, two chemistry and two physics items have been added in rota- 
tion to the 2k items in the second test of Population IV. Each student 
in this population will therefore respond to six IVS items, but for each 
group of six students, 36 IVS items, twelve biology, twelve chemistry 
and twelve physics, will be administered altogether. 

?22i2-22-2S22E2i222i2S-£i2«22i2E2 2?-22i e 22 e 

Representations were made by one or two countries that an attempt 
should be made to assess the students' ability to understand the nature 
and methods of science as distinct from the purely cognitive aspects of 
science. With this end in view, a test (SST 1 - Supplementary Science 
Test 1), which drew very heavily on the TOUS tests devised at the Uni- 
versity of Chicago, was compiled and pre-tested in September/October 1968. 

Comments were received from eleven countries and full pre-test re- 
sults, including item analysis, from eight. On the strength of this 
information it was decided to include a separate test on ^'Understanding 
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the Nature of Science” for completion with the questionnaires on the af- 
fective measures by Populations II and IV. The tests for both Popula- 
tions will consist of 15 items, for which 20 minutes of testing time j 

will be allowed. Seven items will be common to both Populations. 1 

Tests on grac tical abilitiea_in_science j 

It was stated earlier that major differences occurred between 
countries with regard to the place accorded to practical work in the 
laboratory or field. Many of the new developments in science education 
are concerned with the question of the nature and extent of the first- j 

hand experience that is desirable during the study of science at school. 1 

One of the mo6t important hypotheses to be tested by the science pro- 
ject is in fact, that science teaching methods baaed upon actual en- 
quiry by thoroughly sound scientific methods achieve better results, 
even measured in cognitive terms, than traditional methods. It had 
been hoped that suitable items for measuring these abilities would have 
been found amongst those submitted by the various countries and a strong j 

appeal appeal was made for them by the Science Committee. Since none 
were forthcoming, a grant was made jointly by IEA and the NFER England j 

for a study of the problem, and the production of suitable items. This ; 

study was based at the University of Leicester, England. As a result 
two tests SST 2 and SST 3 (Supplementary Tests 2 and 3) were devised j 

and pre-tested together with SST 1 in September/October 1968 . Some of 
the items are of a paper and pencil type; others require a minimum of 

apparatus. 1 

! 

Results of the pre-testing of these "practical" tests were received 
in time from three countries only, although comments were received frcmi | 

sever al others. It seems clear that some countries do not attach as much 
importance to practical work as others, while the administration of tests | 

requiring even a minimum of equipment and laboratory space presents dif- J 

ficulties that few countries expressed willingness to undertake. In | 

consequence, it was decided to incorporate ten "pencil and paper" items 
aimed at measuring the results of practical experiences in each of the 
two cognitive tests for Population II and twelve similar items in test j 

IVA for Population IV. Eight of thes® items will be common to both 

Populations. . 

In addition, five "practical" laboratory tests involving only a : 

small amount of very simple equipment will be offered as a national op— | 

tion. These are all planned for one and a half hours in the laboratory 
although it is expected that the actual manipulations and recordings | 

will be completed within an hour. The intention is to remove from the 
fetudent all sense of undue pressure. One of these five tests will be 
in science, equally divided between biology, chemistry and physics, for j 

Population II. The other four, in science, biology, chemistry and 
physics respectively, will be for Population IV as school options within 

the national system. 

A s ummar y of the final science tests is given on the next page. 
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A Summarjr_of _the_Final_ScienceJTest8 



if 




















I 



Population 


Name of Test 


No. of Time 
Items Minutes 


Comments 


I 


Science I A 


20 


30 


Most of the knowledge and 
understanding tested is 




I B 


20 


30 


not likely to be the re- 
sult of specific science 
teaching but of the aware- 
ness of the opportunities 
provided by the environ- 
ment in general elemen- 
tary teaching. 


II 


Science II A 


4o 


60 


Both tests contain "paper 
and pencil practical" 




II B 


ko 


60 


items. 


IV 


Science IV A 


36 


60 


IV A will contain "paper 
and pencil practical" 
items. 




IV B 


24 


60 


Bach set of six rotating 






+6 rota- 




items will contain two 






ting IVS 




items each on biology. 






items 




chemistry and physics and 
will be answered by one 
student in every group of 
six students. 


IVS 


Science IVS 






A student in a country 




Biology 


40 


60 


which has elected to take 




Science IVS 
Chemistry 


40 


60 


these tests will take 
either one or two of them 
at either his own choice 




Science IVS 






or that of the school. 




Physics 


40 


60 





It is suggested that countries intending to test Population III 
should use tests II A and II B» 



As a national option the following "practical" tests are offered: 



! 



I 



V-- 

1 
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2.5 S tudent , Teacher_and_S chool_Ques t ionnaire 

Concurrently with the production of the cognitive measures ques- 
tionnaires were developed which will provide some of the informa 
tion required when the hypotheses are being tested. Those relating 
specifically to science, ST 1 and 2 - Science and Teacher Question- 
naire - Science are shown in Volume II, Booklet 4. 



2 . 6 General_Appraisal 

The science part of the project has reached the present stage as a 
result of the very great efforts made by the National Centers and the Science Com- 
mittee backed by the very efficient Secretariat, and partly by the cal- 
culated risk taken by the Science Committee in putting out for comment 
and trial materials with which they were by no means satisfied, but 
which they felt to be potentially satisfactory. Criticism was often 
harsh, occasionally unhelpfully destructive, but sometimes constructive 
and valuable. In spite of the initial difficulties an enormous amount 
of work has gone into the production of these instruments and there is 
now a general feeling that the instruments in science are very adequate 
for the purposes for which they have been produced. 
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CHAPTER 3 



Reading Co^rehensjLon 



3,0 Introduction 



The belief that it might be reasonable to include reading compre- 
hension as one aspect of further cross-national surveys of educational 
achievement stemmed from the initial pilot international study (Foshay 
et al , . 1962 ). In this exploratory study, a reading comprehension test 
was included as one element in the test battery. It was then recog- 
nized that translation into eight different languages might give rise 
to severe problems, but the task was undertaken. As the results were 
examined, it was found that be tween-country variance in mean score was 
relatively small, and furthermore the relative difficulty of single items 
remained quite stab?,© from one language to another. The average corre- 
lation of item difficulty indices from language to language was approxi- 
mately .90. An item that was hard in one language tended to be hard in 
others. 



Further work at the end of secondary school level (Kumbaraci, 1966), 
comparing difficulty in English and Turkish, tended to support the earlier 
findings. However, in this older and more select group, the correlations 
were about .70 rather than .90. Also, comparison of items, with respect 
to their ability to discriminate high-ability from low-ability students, 
indicated that this aspect of the items was quite unstable from the one 
language to the other. 

All in all, the results on using reading tests as translated seemed 
sufficiently promising so that further work in this area was proposed. 

The further work would use the larger and more carefully selected sam- 
ples that were characteristic of the IEA mathematics study (Husen ed. , 
1967 )« and would gather extensive data on individual background, school 
characteristics, and national educational programs in order to study 
relationships between these factors and reading achievement. Therefore, 
a Reading Comprehension Committee was set up within IEA. 



Original members of the Committee were: 

Prof. Robert L. Thorndike , Teachers College, Columbia University, 
U.S.A,, Chairman. 

Dr. Lily Ayman, Teachers College, Tehran, Iran. 

Mile. Francois© Bacher, Institut National de 1 * Orientation Pro- 
fessionnelle, Paris, France . 

Nr. Alan Brimer, Director, Educational Research Institute, Insti- 
tute of Education, University of Bristol, England. 

Mr. Alan Murphy, Institute of Education, University of Ibadan, 
Nigeria. 

Mr. Murphy found himself unable to continue after the initial meeting cof 
the Committee, and his place was taken by Dr. Jan Vastenhouw, Institute 
of Education, University of Amsterdam, The Netherlands. 
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3.1 Preparation of Rationale 



| 

l Initial planning for the study of accomplishment in reading com- 

f prehension in different countries was carried out at a meeting of the 

| IEA Reading Comprehension Committee in Hamburg in November, 1966. At 

\ that time, a statement of rationale for and procedures tentatively to 

I be carried out in a reading comprehension study were developed. This 

rationale was revised at a June 196? meeting of the Committee on the 
I basis of comments received from different national groups and, in its 

revised form, is attached to this report as Appendix E. 

As indicated in Appendix E , several considerations led the Com- 
| mittee to focus on the cognitive aspects of reading rather than the 

I aesthetic and affective, and upon expository rather than narrative raa- 

[ terial. Practical realities restricted the assessment to paper-and- 

| pencil group testing, and this, together with the focus of interest of 

those working in other subject areas, ruled out testing of children be- 

| fore the age of 10 years. 

| 

The above constraints ruled out much of what would have been most 
| interesting in the field of reading, to wit, the early development of 

I reading skills. The focus was upon comprehension by relatively mature 

( readers, rather than upon the initial stages in development of decoding 

r skills. 



| It was initially proposed that we test both ability to get informa- 

f tion from a text when the text is present to be referred to and ability 

to answer by recall questions based on a text that had previously been 
. studied. The second type of testing was abandoned in the final instru- 

ments, primarily because of lack of time— both lack of time to prepare 
and try out the test materials and lack of testing time upon the occa- 
sion of the final test. 

\ 

| Word knowledge was seen as both a condition of and an aspect of 

[ reading skill, so a word-knowledge test was developed as described more 

fully below. A reading speed test was also developed, since speed seemed 
a relevant second dimension of reading competence. 



The statement of rationale was sent out to the National Centers 
to elicit comments from the national Reading Comprehension Committees. 

In addition to being asked to comment on the general plan and rationale 
for tho work in reading comprehension, each National Committee was in- 
vited to submit reading passages, especially ones with test items al- 
ready available, that might feed into che final test of comprehension 
as the test was developed. These materials were intensively reviewed 
at a meeting of the Reading Comprehension Committee in Paris in June, 
1967. Passages were rated as to their apparent difficulty by members 
of the Committee, and sorted into seven levels that were described as 
8-year-old, 10-year-old, 12-year-old, 14-year-old, l6-year-old, 18-year- 
old, and 18+ • Bie designations represented the age level at which it 
was judged the passages would be appropriate for the typical student. 
However, the judgments were intended to represent only a rough prelimin- 
ary grading of the passages. 
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3.2 Preparation of Materials 



A number of passages that seemed to the Committee members to be 
suitable in style and content, and not to be peculiar to any one culture 
or country, were selected as possible materials for the test. These 
passages were reproduced and circulated to all of the countries for com- 
ment, and only those passages that were almost universally considered 
to be acceptable for inclusion were retained for further development. 
During the summer of 1967, an extensive assortment of new test items 
was prepared, primarily by the chairman of the Reading Comprehension 
Committee, to supplement those that were already available for the pas- 
sages. (Many existing items were modified and edited to cast them into 
a multiple-choice format that would permit machine scoring. ) These 
items' were subsequently sent to all National Committees for comments 
and suggestions and for the supplying of additional items wherever a 
National Committee cared to make such a contribution to the enterprise. 

Taking account of the comments and suggestions of National Com- 
mittees, and including items provided by these groups wherever possible, 
six try-out test forms were prepared for each of the populations to be 
included in the IEA study to wit, 10-year-olds, l4-year-olds, and stu- 
dents in the final year of secondary education. Each 10-year-old try- 
out test included three passages, one that had been rated as an 8-year- 
old passage, one a 10-year-old passage, and one a 12-year— old passage* 
The try-out tests for Population II (14-year-olds) contained four pas- 
sages spanning the 12- , 14-, and 16-y ear-old levels. The end-of- 
secondary test also included four passages, but at the 16, 18 , and 18 + 
levels. Thus, the test at each level had one passage in common with 
the next level. All told, 50 different passages and 445 different items 
were tried out. 



3*3 Pre-Testing 



Tests were distributed for try-out approximately in March, 1968, 
and try-outs were completed before the end of the 1967-1968 school year. 
The amount of material to be translated and tried out was more than 
could have been handled in any one country, so the six forms of the 
tests were divided into three sets of two forms each, and each parti- 
cipating country was asked to pre-test one-third of the material. The 
schedule called for each test to be tried out in four countries, one 
an English-speaking country and the other three each representing a 
different language. It was hoped that in this way each test would have 
been pre-tested in enough different languages to make it possible to 
eliminate items that were discriminating in only one or two languages, 
and that the items that were retained would prove to be discriminating 
in the remaining languages as well as in those in which they were tried 
out. For most tests, data were available as planned for four countries, 
but difficulties arose in relation to some of the try-outs so that a 
few passages were tried out in only two or three countries. 
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3.4 Preparation of Final Tests 
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Item statistics on the try-out data were returned to IEA head- 
quarters at Hamburg* where they were collated and made available to the 
chairman of the Pending Comprehension Committee who in turn organized 
them in relation to specific passages and distributed them to the mem- 
bers of the Reading Comprehension Committee. Committee members were 
asked to indicate which passages should certainly be kept* and which 
should be the ones to be eliminated. Within passages that might be 
kept they were invited to nominate items for retention and for rejec- 
tion. Final selection of the passages and items became the responsibil- 
ity of the Committee chairman. 



Passages were selected to provide a suitable range of difficulty 
for the population in question. A second consideration in 
of passages was the number and diversity of items showing suitable dif- 
ficulty and discrimination. Finally, the attempt was made to get vari- 
ety of content and treatment. Items were retained that showed acceptable 
difficulty and discrimination indices* and within these limits an attempt 
was made to include items covering as wide a range of reading skills as 
possible* i.e., items dealing with specific factual details, with the 
main idea, with inference beyond the literal content of the passage, with 
the author* s point of view and purpose, and with the author s writing 

techniques. 



Final tests were prepared as follows: For Population I ( 10-year- 

olds) two tests each planned to have a time allowance of 25 minutes, 
covering 8 passages and including a total of 45 items; for Population II 
(li-year-olds) two tests each planned to have a time allowance of 45 
minutes, covering 8 passages and a total of 52 items; for Population IV 
(end of secondary school) two tests each planned to have a time allow- 
ance of 50 minutes, covering 8 passages and a total of 54 items. 



3. 5 Reading Speed 



The original rationale for the reading appraisal included a pro- 
posal to assess reading speed as well as reading comprehension, since 
speed is a relatively independent aspect of reading skill that has some 
importance as an academic accomplishment. Assessment of reading speed 
has always presented a number of technical problems and no fully satis- 
factory technique for this type of assessment has been developed. The 
plan originally proposed by the Reading Comprehension Committee was to 
appraise reading speed by presenting passages into which words had oc- 
casionally been inserted that spoiled the sense of the passage and had 
no relationship to what went before or after. The examinee was to be 
instructed to underline or cross out these words as he came to them, and 
the number of such cross-outs in a closely-timed test would indicate the 
speed at which he was reading. 



One problems with this technique is that it is a somewhat unnatural 
one, implying a set to proofread rather than to read for sense, and 
another difficulty is that if the material is of a fairly high level. 
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f the speed may be very much slowed down by comprehension difficulties* 

Somewhat unilaterally, the chairman of the Heading Comprehension Com- 
mittee modified this original pattern to one that has been widely used 
with young children in tests of reading speed in the United States. 

Hhis procedure involves providing a series of relatively short para— 

I graphs, each of which ends with a question and is followed by three 

words, one of which is to be underlined as answering the question. The 
i paragraphs have to be constructed in such a way that the most economical 

and efficient way of answering is to read the whole paragraph, rather 
S than reading merely the question. When this is the case, the process 

is a re as onably natural one of reading a sentence or two followed by a 
question and then responding to that question. 

t In the materials that were prepared for this purpose in the IEA 

study, the sequence of paragraphs constituted a single continuous story. 
The attempt was made to write the material at a very low level of dif- 
ficulty, so that the measure would be that of the fluency in mechanics 
[ of reading, material of a very simple nature. Two stories were/ prepared 

and were distributed to the national groups for comment and subsequently 
jj for try-out. The materials were administered with the instruction for 

1 each child to read as rapidly as he could, while still getting the in- 

f formation needed to answer the questions. The child was instructed to 

j mark clearly where he was at the end of three minutes and again at the 

| and of six minutes so that it would be possible to get two distributions 

of items attempted and provide two marks to guide the final decision as 

to the optimal testing time. 

I 

In addition to getting data on the timing of the passages, each 
country was asked to tally the number of errors on each of the items in 
the stories so that items that were ambiguous or too difficult could be 
} identified and modified. It turned out that one story had a number of 

f rather ambiguous items and items that presented difficulty to children 

whereas the other had relatively very few. On this basis, the second 
story was chosen for the final testing and was edited to try to mini- 
| mize those difficulties that were in fact found in it. The first story 

| was also edited to eliminate as much as possible the ambiguities and 

| difficulties, with the intention that it would be used as a practice 

story. The time limit that seemed appropriate for final testing with 

l the speed test was four (4) minutes. 

\ 

: Certain questions arose in discussing the processing of the data 

[ from the speed test. These have to do with a practical method of scor- 

| ing, since the test did not use a separate answer sheet and scoring for 

correctness would be quite laborious. It was initially intended that 
[ the test would be scored merely in terms of the last item attempted, 

I since it was intended that there would be a very low error rate on tha 

test. In practice, it turned out that in a few of the countries the 
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story wbs a good deal nor© difficult than had been expected and the 
error rat© was fairly substantial. The problem of how to adjust a 
I speed score for errors when these are fairly frequent is one that is 

not easily solved. Conceivably, one could score merely the number of 
| correct responses, or one could even introduce a penalty for errors so 

that the score for effective reading speed might be the number right, 
minus some fraction of the number wrong. Any of these scoring proce- 
| dures in troduces very serious practical scoring difficulties, and this 

problem remains not fully resolved at the present time. What seems a 
I practical compromise is to count errors on only the first page of the 

test, thus getting an error score, but quite an unreliable one for each 
individual. The speed score would be the last item attempted. Having 
both scores available, one could study them separately or in any weighted 
combination. Hand-scoring only the first pas© for errors should go very 
quickly and not prove burdensome for National Centers. 



3.6 Word Knowledge Test 

A third component that was proposed in the original plan for the 
reading test was a test of word knowledge. This was seen less as an 
inherent part of the reading test than as a control variable that might 
be useful and relevant in relation to all the tests of the IEA study. 



As something of a tour de force , the Reading Comprehension Com- 
mittee undertook to try to produce a common test to be used across 
languages in all of the different participating countries. 



The item format that was selected was that of word pairs where the 
words of a pair must be judged to be approximately either synonyms or 
antonyms, that is, either nearly the same in meaning or nearly the op- 
posite in meaning. It was thought that this item format would be easier 
to translate into different languages than the conventional multiple- 
choice format with a number of wrong options. Here it was felt that 
the attractiveness of wrong options would vary so much from country to 
country that it would be extremely difficult to get equivalent items 
from one language to another. 



In order to accumulate a pool of items that might be used for this 
study, National Committees in each country were invited to submit sets 
of word pairs ranging from easy to difficult, providing them to the Read- 
ing Committee in their English translation. A list of some 300 word 
pairs was assembled. The difficulty of these in English was judged by 
several individuals at Teachers College, and the words were arranged into 
10 classes with respect to their judged difficulty. The set of words 
was then circulated to each of the National Committees, and each Com- 
mittee was asked to indicate which of the pairs would be either impos- 
sible or difficult to translate into a pair of corresponding difficulty 
in the language of the country. 
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The data from the above assessments were assembled and tabulated, 
and all word pairs were dropped from further study in which more than 
one country indicated difficulty in the matter of translation. The re- 
maining word pairs were assembled into three try-out tests, one for 
10 -year-olds, one for l 4 -year.-olds, and one for end of secondary school 
and were circulated to the participating countries for try-out in their 
school systems. Some of the items in adjacent tests were common to 
permit scaling items from successive test levels on a common scale. 

The original hope and plan in designing the vocabulary test was 
that there would be among the rather large pool of items tried out a 
sub-set of items that were of comparable difficulty from one country 
to another. The problem becomes one of defining comparable difficulty 
in different countries using different types of samples and different 
languages. The assumption on which the attempt to scale difficulty 
across countries was based was that on the average for the whole pool 
of item« they would be about equally difficult in English and in the 
language into which they were being translated. The subsidiary expec- 
tation was that the correlation of item difficulty from one language 
to another would be positive and fairly high, though obviously far from 

perfect. 

Data were reported to IEA in the form of percent passing a given 
item. In some countries, item vs. test correlations were also reported, 
making it possible to identify items that were not differentiating be- 
tween the more and the less able pupils. 






k 

a 

I 

I 



I 



I, 

h 

I 

I 

I 

*V 



!• 

Is 



Items were scaled for difficulty, first within each age sample, 
then across age samples for a country, and then in relation to the U.S.A. 
as a common anchor population. The sequence of steps is outlined below: 

(1) Proportion "knowing” an item was defined as proportion correct 
minus proportion wrong (R-W) . 

( 2 ) The proportion established in (l) was converted into a devia- 
tion score on the baseline of a normal curve . 

( 3 ) Using those words that were common to the Population I and 
Population II tests, the difference in means and ratio of standard de- 
viations in the two groups was estimated. 

(4) Scale scores for the Population II items were converted to the 
Population I scale by the appropriate linear equation. 

( 5 ) In the same manner, Population IV scale values were converted 
to Population II and finally to Population I scale values. 
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(6) Scale values as determined in (5) for Country X were plotted 

| against U.S.A. scale values, and the means, standard deviations, and 

intercorrelation computed. 

(7) A conversion equation was applied to the Country X data to pro- 

\ duce the same mean and standard deviation as those for the U.S.A. 

I 

I As the data were received and examined, it seemed rather doubtful 

1 as to whether either of these original expectations were borne out. In 

; particular, there were a number of the translation languages in which 

[ the items showed a very limited spread of difficulty. Generally speak- 

| ing, the range of difficulty was greater in English than it was in the 

f languages into which the words were translated and this kind of a re- 

gression towards mediocrity was larger in some languages than in others. 

I 

I It also appeared that the items tended to get systematically easier 

| in translation, though this cannot be tested in any rigorous sense be- 

cause one doesn't know much about the comparability of the samples from 
| the different countries. It certainly did appear that the items which 

| had been very difficult in English lapsed into being only moderately 

I difficult in many of the translations, and that there were relatively 

j few instances of the reverse, where an item that had been only moder- 

| ately difficult in English became very difficult in one of the other 

| languages. Furthermore, the average correlation of item difficulty from 

language to language, even including a range of words from those that 
5 were quite easy for 10-year-olds to those that were quite difficult for 

| pupils at the end of secondary school (in English in any event), was 

only about .4 or .5. As a result, there was very large scattering of 
item difficulty, and it was hard to find a sub-set of items that were 
| very nearly the same in difficulty across all languages. An attempt 

l was made to select the best sub— set possible in the sense that items 

were chosen to try to balance difficulty from one language to another 
and items were chosen where the spread of difficulties was relatively 
{ small compared with others of the items. However, it must be admitted 

that the final result is one in which the equivalence from language to 
language is suspect at best. 

Thus, to summarize, the following tests have been produced: 

Population I Reading Comprehension Part I 

Part II 

Reading Speed 
Word Knowledge 
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Population II Reading Comprehension Part I 

Part II 

Reading Speed 
Word Knowledge 



Population IV Reading Comprehension Part I 

Part II 

Word Knowledge 

The word knowledge test will be administered as part of the General 
Questionnaires in all countries* 

The final instruments are to be found in Booklets 3 and 4 of 
Volume II* 
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4.0 Introduction 



The IEA Study of achievement in literature has grown out of an 
effort begun in 1962 by Alan C. Purves, then at Barnard College, 
Columbia University, and A. W. Foshay, of Teachers College, Columbia 
University. These two colleagues considered whether the responses of 
students to literature could be analyzed cognitively; as a result Pro- 
fessor Purves developed a schema, "The Elements of Criticism and 
Interpretation" which sought to represent the field as a cognitive map 
(see Appendix F-l) . 

The schema was first subjected to critical scrutiny by a panel of 
well-known U.S. literary scholars and critics. Having survived this 
examination, the schema was further tested by using it to code brief 
essays written in response to an identical short story (translated as 
necessary) by secondary school students in Germany, Belgium, England 
and the United States. Once more, the schema survived; three independ- 
ent readers coded the sentences in these essays with two-way agreements 
on 80% - 90% of the students 1 sentences, and three-way agreements on 
60% - 70% of the sentences. 

At this point, Professors Purves and Foshay proposed formally to 
the IEA Council that literature be included in Phase II of its work. 

The proposal was eventually accepted. An International Committee of 
five (see Vol. II, Booklet 6, p. 23) was appointed, and the IEA Council 
members from seven countries agreed to participate in this stage of the 
work. The countries were: Belgium, Chile, England, Iran, Italy, Sweden, 

and U.S. A. 



In each of the participating countries a National Committee was 
appointed (see Vol. II, Booklet 6, pp. 65-99), consisting of scholars 
and educational leaders in Literature in the mother tongue. 



4.1 Preparation of Rationale 



The IEA International Committee meeting on literature prepared a 
draft rationale for the study in 1966 (see Appendix F-2) . This 
statement was circulated to the seven participating centers for criticism, 
and also to provide for them a framework within which they might carry 
out their first task: to prepare a statement of the goals of the teach- 

ing of literature in each country. 
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These statements of goals were returned to the International Com- 
mittee, and became the basis for final decisions concerning the areas 
of achievement in literature to be measured.* 

When the national statements of goals had been analyzed, it became 
apparent that there are, broadly stated, four categories of goals of 
instruction in literature: 

1. Acquaintance with the principal literary works and the his- 
tory of literature on one's own country and language. 

2. The ability to respond skillfully and appropriately to liter- 
ary works of several genres, (short stories, novels, lyric 
poems, epic poems, dramas, etc.). 

3. The development of favorable attitudes towards serious liter- 
ature, and a disposition to partake of it voluntarily. 

4. The expression of a preference for literary works acknowledged 
to be of high quality. 

4.2 The Selection of the Goals 



Two of the four goals do not lend themselves to international 
testing. Goal number one - acquaintance with one's literary heritage, - 
is important, but unmanageable for international purposes. It is con- 
ceivable that some metadimension of achievement in this area could be 
identified and measured, but the development of such a construct seemed 
to the Committee so complex as to be beyond particability in the present 
project. 

Goal number four - expression of a preference for literary works 
of high quality - could be measured with some difficulty, because prefer- 
ence refers both to content and style, and it is hard to hold one dimen- 
sion constant. Moreover, on an international scale such a measurement 
runs the risk of being offensive. Such a test would have to venture 
into the highly personal world from which one's tastes arise. Moreover, 
the national committees and other advisors considered that such an at- 
tempt might be perceived as an effort to legislate taste, the last thing 
anyone wants. With some regret, therefore, the International Committee 
decided not to measure the achievement of this goal. 



•Note: the US document reporting the goals of teaching literature was 

particularly thorough. It has achieved a considerable amount of atten- 
tion among teachers of literature in the US. Requests for copies of 
this report have been received from several states, and it provided the 
basis for a chapter by Professor Purves on evaluation in literature, to 
be included in a forthcoming book, edited by Professor B. S. Bloom. 
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The remaining two goals - the ability to respond to literature, 
and the development of favorable attitudes toward serious literature - 
therefore became the basis on which the committee proceeded to the de- 
velopment of tests. The choice was defensible, because in each of the 
countries, these two goals were considered central. Eie first of them 
can be called ’’literary understanding,’’ which in turn has both an 
achievement dimension and a preference dimension (understanding *jas 
slightly different definitions in different countries and n ^ n J d J“? ren l r 
critical schools within countries). The second can be ca * led llte rary 
attitudes.” In order to develop instruments appropriate *° th * 6 °?t s ' 
the committee had to start from scratch. Almost no extended attention 
has been given to such measurement, even within nations. A cross- 
national attempt of this kind has never been undertaken, f^ch instru- 
ments as exist do not, in the committee's judgment, deal with these two 
goals with sufficient directness. At every step in the development o 
these instruments, therefore, the committee has had to break new ground. 

4.3 Pogulations_to_be_Tested 

We will study IEA Populations II and IV; all l4-year-olds, and 
all students in tZ final secondary school year. In addition, students 
in Population IVS, who are specializing in literature, wiUbe a 
write essays in response to a short story by Hemingway. IEA Population I 
will not be included in this study. In most countries there is ^o de- 
liberate instruction in literature in the elementary school, from which 
this population of 10-year-olds would be drawn. 

It should be emphasized that the collection of essays from Popula- 
tion IVS, is considered by the committee to provide a very valuable col- 
lection of data, which can be used not only for the present study, but 
also for future studies of the nature of literary responses among young 
people? This^ollection of student essays will form a kind of data bank 
for such studies during the years ahead. 

4.4 The Ins trument s _to_be_us ed 

As finally developed, the instruments deal primarily with the na- 
ture and the quality of the student's response to some actual texts, 

52 Zs preferSS Response (as related to "Elements of Writing about 
Literature") and with his attitude towards literature. 

Practical considerations led the committee to develop a set of 
measurement tasks that can be completed by most students in two hours. 
Accordingly, the following agreements were ultimately developed. 

Each student will read and respond to two short stories. 
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2. A total of four short stories will be used. One of these will 
be a "common test," administered to all students in both popu- 
lations. The other three texts will be rotated among students 
in Populations II and IV. 

3. Two other kinds of instruments were required: an inventory 

of attitudes towards literature, and a "response preference" 
test, based on Professor Purves* "Elements." 

4.5 Preparation of Tests 



The development of the actual tests required approximately one and 
a half years. Since the nature of the IEA project made it impracticable 
to read and analyze essays written in response to texts, the committee 
considered alternatives. Professor Britton, of the Committee, had de- 
veloped a technique of linked open-ended questions, which led the stu- 
dent through some elements of a given text, then asked that he interpret 
them. These open-ended questions seemed highly valid, while also cor- 
responding closely with typical kinds of instruction in classes. How- 
ever, such responses gathered from large populations internationally 
would not be manageable. 

The committee therefore undertook a "crossover" study, in which 
the question at issue was whether multiple-choice items could elicit 
essentially the same responses from students as the open-end questions 
in which the students composed their own responses. 

The crossover study (see Appendix f- 3) was carried out between 
January and June, 1968, based on samples of approximately 4 00 Popula- 
tion II and IV students in England, and the same numbers in the US. 

The principal findings of this study were that the two test forms - 
open-ended and multiple choice - correlated at approximately +.70 which 
was high enough to justify proceeding with the multiple choice form. 

Some other questions also arose. Given the assumption that re- 
sponses to literature were at the heart of the matter, what literature 
should the IEA students respond to? Clearly, if a way could be found 
to manage it, lyric poetry should be included, for response to lyric 
poetry is the most "literary" of responses. 

The most obvious problem in international testing of responses to 
lyric poetry is that of translation. A lyric poem in translation is 
actually a new poem created on the basis of the original. No one knows 
what changes, both obvious and subtle, have taken place. Since we were 
to deal with languages and cultural differences as diverse as Farsi, 
Italian, and American, it was apparent that translated lyric poems 
could not be treated like a common text. 
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What of equivalent lyric poems? Could poems from similar periods 
and with similar themes, be used? To examine this possibility, a small 
comparative study was undertaken (see Appendix F-3) • The conclusion, 
not unexpected, was that understanding of literature was text bound, so 
that more than one text would be needed to indicate a student’s general 
understanding, but that two texts, no matter how similar on the surface, 
were not similar enough to provide comparability. Regretfully, the com- 
mittee abandoned further attempts to deal with cross-national, cross- 
language comparison of responses to lyric poetry. 

J The experience of the study participants to this point had, how- 

ever, made it evident that prose passages can be translated with enough 
| eaui valence to permit fruitful comparisons of responses. The committee 

therefore selected from a long list of possibilities, six prose narra— 
l tives, five short stories, and a selection from an autobiography. The 

I selections were: 
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My Childhood , by M. Gorki (Russian), a selection 

The Use of Force , by William Carlos Williams (U.S.A.), a short 

story I 

Just Lather. That’s All , by Hernando Tellez (Spanish), a short 

story J 

2he Sea. by Ana Maria Matute (Spanish), a short story 1 

ft] 

The Man by the Fountain , by George Hebbelinck (Flemish), a short j 

story l 

I 

I See You Never , by Ray Bradbury (U.S.A. ), a short story J 

The committee developed both open-ended questions of the. Britton type | 

and parallel multiple— choice items for all six of these texts. Not 1 

all the participating countries tested all the texts, but enough in- 
formation was collected to make it possible to be selective of both j 

items and texts. ] 

By October, 1968, the prepared test items had been pretested in | 

Belgium, Chile, England, Iran, Sweden, and the US. Local reasons pre- 
vented pretesting in Italy. 

\ 

Pre-test data indicated for each population in each country and J 

for each item, the following: f 

I 

a. difficulty (the percent choosing preferred response) j 
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b. discrimination (point biserial correlation of preferred re- 
sponse with total score) 

c. percent choosing each of the distractors 

d. point biserial correlation for each distractor with total score. 

The data thus collected formed part of the basis for decisions to 
keep, revise, or eliminate items. In addition to this information, the 
committee had access to the responses of students to Professor Britton's 
open-ended questions. Wherever possible, the revisions of test items 
were based on student constructed responses. 

Figure 1 illustrates the kind of item analysis information the 
committee worked with. The item deals with a short story called "The 
Sea," which will be given to all of the students in the study. In 
column 1 (DOT) the percent of students choosing the preferred response 
is given. In this instance, 96#, 9*# and 86# of the students chose the 
preferred response. The item is too easy. 

In column 2 ( DISC ) is given the point biserial correlations between 
the preferred response and total score for the students. In this in- 
stance, the discrimination of the item is low, but acceptable. Columns A, 
B, C, and D indicate the percentage of the population ‘choosing each of 
the options. Option B, "he wants to wash his ears," did not function, 
nor did option D function well. Oiese two options were therefore re- 
vised to make the item more difficult and also to make it more repre- 
sentative of what is important to the understanding of the story. Sim- 
ilar information was collected from all of the countries for all of the 
items, as has been indicated. In revising the item, the Committee con- 
sulted student constructed responses to the same stem in open form. The 
item as it finally appeared was as follows: 

"Which of the following best explains why the boy first said, "I 

want to see how high the sea will come up on me" (line 12)? 

( ) A. He wanted to cover his body with the sea. 

( ) B. He wanted to see how tall he was. 

( ) C. He wanted to trick his mother. 

( ) D. He wanted to show his mother he was not afraid of the sea. 

Criteria for the Selection of I tems 

In selecting items to be included, the committee applied two main 
criteria, in addition to the usual criteria of indexes of appropriate 
difficult and discrimination: 
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FIGURE 1 

LITERATURE POP. IV SUB-TEST JB ITEM BJ TOPIC OBJ. 

3. Which of the following best explains why the boy first said, 

”1 want to see how high the sea will come on me" (lines 11-12)? 



( ) 


(A) He wanted 


to be covered by the 


sea. 










( ) 


(B) He wanted 


to wash his ears. 












( ) 


(C) He wanted 


to trick his mother. 












( ) 


(D) He wanted 


to hide from the man who 


cured. 








COUNTRY 


DOT. 


DISC ABC 


D 


E 0 


NR 


COMMENT 


Belgium 


96 


.39 96 0 0 


4 


- 0 


0 


N = 


45 


USA* 


94 


.36 9* 1 3 


1 


- 0 


0 


N = 


91 


Chile 


86 


.33 86 4 5 


4 


- 1 


0 


N = 


121 








MEDIAN 



COMMENTS. 







DECISION: ACCEP T R EVIS E R EJEC T FINAL TEST NO. POP._ j SUB-TEST^ ^ 
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[ 1* -Eh© majority of the items should deal with literary under- 

; standing of the text, as contrasted with reading comprehen- 

sion. By literary understanding is meant -the- ability to re- 
| late part of the work to the whole work and to make reason- 

able inferences concerning the characterization in a story, 
the author’s employment of metaphor, his selective descrip- 
! tion of the environment and the like. 



I 

| 

I 

& 

t 

f 

| 

£ 






2. The items should be true to the text; indirect measures of 
literary understanding are to be avoided in the text-related 
items. For example, negative items (e.g., "which of the fol- 
lowing four interpretations does NOT fit • • • ”) too often 
appear to depend on general intelligence, or acuity in read- 
ing the item stem, at least as much as they depend on liter- 
ary understanding. Therefore, except in those cases where 
the negative stem represents the author’s original intent, 
these items were dropped or rendered into positive (declara- 
tive) form. 



h 




The tests finally developed are as follows: 

1. Response preference inventory (20 kinds of responses to each 
text are offered; the student selects five from these, and 
assigns a rank to one of these five). 

2. Literary interest profile (included in student questionnaire) 

3. Attitude towards literature as a field of study (included in 
student questionnaire). 

4. Understanding and Analysis: four texts. 

a. El Mar , by Ana Maria Matute. Eighteen multiple-choice 
items. A text to be given to all students taking part 
in the study. 

b. The Use of Force , by William Carlos Williams. Twenty-one 
items. 

©• De man naar de fontein . by Georges Hebbelinck. Twenty- 
one items. 

d. I See You Never , by Ray Bradbury. Nineteen items. 

Note: Texts b, c, and d are to be rotated between students within a 

population. 
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5* The student evaluation of each of the texts he reads, given 
on a 6-point scale, from +3 to -3* 

6. For Population I VS only, an essay in response to The End of 
Something by E. Hemingway* (The relatively small number of 
essays so collected will be read and analyzed centrally by 
trained readers, using the completed code from the "Elements 
of Writing about Literature." They will be kept for use by 
the international research community*) 

The information to be obtained in this study may be classified 
into input, process and output variable, with the understanding that 
some may function both as process and as output* The measure of 
literary understanding, of attitude toward literature, of literary 
preference, and of response preference may all bo considered as output 
of instruction in literature, since they reflect important goals of 
instruction in this field* As is true in all the subject areas in the 
IEA study, such variables as sex, age, rank in class, socio-economic 
status of the family, membership in one or another ability group or 
stream will be related to these output variables. Similarly, the rela- 
tion between the dependent variables and such environmental variables 
as the kind of instructional materials used, the length and frequency 
of class periods, the kind and amount of independent work (including 
homework), the frequency and nature of external examinations, can also 
be examined* 

Perhaps more interesting than these is the possibility of sub- 
dividing the population according to patterns of literary understand- 
ing (taken as sub-scores), seeking to predict attitudes, school environ- 
ment and other matters* 

It is intended that the data be analyzed to reveal patterns of 
literary understanding and behavior, as contrasted with simple scalar 
measures* For example, the responses to the texts can be examined for 
patterns of error as well as for patterns of success* Broad human 
themes are treated by these texts s love, death, aging, violence. The 
treatment of these themes by students responding to the texts will have 
very considerable intrinsic interest, being related to country, school 
type, age, sex, attitudes, etc* 

One interesting consequence of the work so far has been the pub- 
lication of some aspects of the research that have gone into the prepara- 
tion of these tests. "The Elements of Writing about Literature" by 
Professor Purves and Victoria Rippere was published as Monograph No* 9 
in its research series by the National Council of Teachers of English 
(USA) in 1968* This monograph has been translated into French for pub- 
lication in 1969 in Paedogogica Exnerimentalis in Belgium. The cross- 
over study described earlier will be reported by Dr. Bruce Choppin in 
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Vol. XV No. 2 of the International Review of Education in June, 1969. 
Several participants in the study report that they have been called 
upon to discuss the technical problems and possibilities of the 
research at professional meetings during the past two years. The 
study appears to be attracting a considerable amount of attention. 



Booklet 3 in Volume II includes the final Literature tests. 
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Frie n £h_a£ a Foreign Language 



5,0 Introduction 



The investigations and discussions leading to the decision to in- 
clude French as a Foreign Language in the I.E.A. study are described 
elsewhere"In"thIs"report7""T3ee CKapter I and Appendix A5. ) 



The original members of the International Committee on French as 
a Foreign Language, appointed in 1966 were: 



Prof, J. B, Carroll (Chairman) 

Dr, John L. D, Clark 
Mrs. Clare Burstall 

Prof, Guy Capelle 

Prof, Wallace E. Lambert 
Dr. H. N. Simai 



(Formerly) Harvard University 
Graduate School of Education, 
Cambridge, Mass, (From Feb. 1967) 
Educational Testing Service, 
Princeton, N.J. 

Educational Testing Service, 
Princeton, N.J, 

National Foundation for Educational 
Research in England and Wales, 
London, England 

(Formerly) Dept, of Romance Lan- 
guages, University of Michigan, 

Ann Arbor, Mich.; Associate Di- 
rector, Center for Research in 
Language and Language Behavior, 
University of Michigan, Ann 
Arbor, Mich. 

Dept, of Psychology, McGill Uni- 
versity, Montreal, Canada 

National Organization of Teacher 
Training and Educational Research, 
Tehran, Iran 



At the initial meeting of the International Committee, held in Hamburg, 
in November, 1966, Professor Lambert was represented by Dr. John Macnamara, 
University of Dublin. In 196?, Dr. Erland Kruckenberg, Larartiogskolan, 
Stockholm, Sweden, attended a meeting of the Committee. 



5. 1 Preparation of Rationale 

The first task of the International Committee was to develop the 
general design of the contemplated studies, detailing hypotheses, vari- 
ables, and target populations. Following are some of the main points 
agreed upon at the initial meeting: 
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1. It would be desirable to have testing done at various educa- 
tional levels, ranging from a population at the 10-year-old level, to 
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the pre-university level. In including the 10-year-old population, the 
Committee was cognizant of the considerable interest that had been 
evinced in foreign language teaching at the elementary s.chool level. 

For example, at two international conferences on foreign language 
teaching at the primary level, that had been sponsored by the UNESCO 
Institute for Education (see Stern, 1963; Stern, 1968), it had been 
recommended that international studies of foreign language achievement 
at the primary level should be conducted. 

2. An attempt would be made to develop uniform tests of French 
proficiency at the various levels, including tests of listening, read- 
ing, writing, and speaking. 

3* The major emphasis of the studies would be on the evaluation 
of actual proficiency in the French language, in both spoken and writ- 
ten aspects. However, to the extent possible, some consideration would 
be given to the evaluation of attainment in "cultural" and "general 
educational" objectives. (Subsequently, it appears to be impracticable 
to include separate measures of these objectives in the testing program 
in view of time limitations). 

4. The Committee was of the opinion that in view of the fact that 
in many of the countries to be surveyed French is only an optional sub- 
ject, the study could better be directed to the investigation of vari- 
ables involved in yielding high proficiency rather than to the investi- 
gation of sheer amounts of "national yield." The comparison of several 
countries with respect to conditions under which good or poor results 
are achieved would, it was felt, make for more interesting and useful 
conclusions than any conclusions based on simple cross-national com- 
parisons . 



The conclusions reached by the International Committee at its November 
1966 meeting were communicated to National Centers (see Appendix G-l). Volumin- 
ous replies and comments were received from many of the National Cen- 
ters. Many National Centers remarked that it was impossible for them 
to state any fixed list of "national objectives," there being a variety 
of objectives represented in different courses and levels of instruction 
in their countries. In general, however, the National Centers approved 
the outlines of the proposals made by the International Committee. 



5. 2 Preparation of Materials 

Bv July 1967 enough information had been received from National 
Centers^ t^make possible a more precise formulation < of the testing pro- 

gram. Accordingly, a sub-committee of the I ^®^ ion ^ 1 Ti^l 
in Princeton, N.J., July 31st - August 4th, 1967 (Carroll, 

Clark). The kinds of tests proposed by this sub-committee 
various populations w©r© as follows * 
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I Population One (Age range: 10.0 - 10.11) 

! 

| Listening Comprehension: Primarily tests in the "four-picture" 

| format ”whereln”the”chlld has to indicate which of four pictures 

| is referred to by a spoken utterance he hears on tape. This. 

I format eliminates the problem of variations in reading compre- 

S hension ability and therefore tends to be a test of pure lis- 

I tening ability. 

Reading Comprehension: Mainly tests using the four-picture 

?ormat7”but”with~tHe”stimulus being a short printed sentence. 

Sgeaking: The test would consist of three sections: Pronun- 

ciation (child imitates spoken utterance and is judged for ac- 
curacy of pronunciation); Structural Control; child answers 
simple spoken questions based on pictures; Fluency: child 

asked to describe in some detail two or three pictures contain- 
ing a variety of situations and elements. 

| PogulationsJTwo and_Three 

i Listening Comprehension: Mainly, multiple-choice questions in- 

I voIv!ng ~spoken ~optIons7 (Later, it was decided that the options 

| would be printed in French.) 

| Reading Comprehension: Printed passages (of various types) fol- 

| lo we d”by "printed "multiple -choice options - all in French. 



Writing: Various types of cons true ted-response items. 

Speaking:. Similar to the test for Population 1. 

Population Four: (Pre-university grade. Both a "survey" test and 

an” rT advanced n test would be developed - the survey test for all 
students to be tested in the IEA sample, and an advanced test 
for students specializing in French. (Population IVS. ) Formal 
tests of listening, reading comprehension, and writing would be 
constructed. A speaking test would also be made available. 



\ 



l A start was made on developing all these test forms required. The 

work on tests for Population 1 was based primarily on a series of tests 
that had already been prepared by Mrs. Burstall for a project she was 
working on in England. The test materials for the other populations 
were for the most part drawn from non-secure item files at Educational 
Testing Service. Some of the materials supplied from National Centers 
were also found useful, and a number of new test items were developed 
by Dr. Clark and Mrs. Burstall. 

I 

\ 

The second full meeting of the International Committee took place 
in London, October 9-13* 1967. The members present reaffirmed the 
Rationale formulated at the November 1966 meeting but took note of cer- 
£aln~revised lists of hypotheses developed by several countries. 
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Uie committee at this meeting also thoroughly reviewed the draft 
tests that had been prepared by Mrs* Burstall and her staff (in England) 
and Dr. Clark (in the USA). Various detailed points were commented on, 
and a number of major deficiencies in the tests were pointed out. In 
several days of work subsequent to the committee meeting, Mrs. Burstall 
and Dr. Clark were able to correct most of the deficiencies (e.g., over- 
emphasis on "rural" content in some of the Population I tests). 

Further, the committee addressed itself to the construction of at- 
titudinal and descriptive scales that would be administered along with 
the tests, and drafted instructions to the National Centers regarding 
the administration of the Tests, translation of instructions, etc., for 
the pre-test phase of the project. 

The tests in the form in which they were prepared just subsequently 
to the October 1967 committee meeting were sent out to National Centers 
for further comment. A number of detailed suggestions were received, 
mostly having to do with phraseology of the test items, and were taken 
into account in the preparation of the final forms for pretesting that 
were then sent out to National Centers in Marbh 1968 along with com- 
plete instructions for administration. 

The test rationale and description of pre-tests is included as 
Appendix G-2. 



5.3 Pre-Testing 

A description of the tests sent out for pre-testing is as follows: 
Poj>ulation_I : 

Liste ning s Two forms were constructed, LI and L2. Each consisted of 
5”sample items and 50 test items (there was some overlap of items 
between the two forms). The items were in a "four-picture" format 
whereby the child had to listen to a spoken utterance on a tape and 
indicate which picture it referred to. Time for administration: 
about 30 minutes. 

Reading: Two forms, R1 and R2, with some overlap between the forms. 

Each” test had 3 sample items, followed by 35 "four-picture" items 
where the child had to indicate which picture a printed stimulus 
referred to$ and 10 multiple— choice items based on short printed 
texts. Suggested time for administration: 30 minutes; however, the 

tests were to be administered without rigid time-limits. 

Sgeaking: One form, SI, consisting of three sub— sections : 

Pronunciation: 21 sentences recorded on tape; the child was to 

imltate”each”sentence, his response being recorded on tape for later 
evaluation of his pronunciation of specific phonemes or intonations. 
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Structural Control: 10 sample pictures; the child was to make a 

on”-sentence”response answering a question (in French) about what was 
shown in the picture. 

Fluency: 3 somewhat more complex pictures; the child was to choose 

one”o?”the pictures to describe in French. 

The Speaking test was to be given to only a sample of children, roughly 
divided among "good," "average" and "poor" learners. Estimated ad- 
ministration time: 15 minutes, individual administration. 

Population_II : 

Listening: 10 four-picture items; 55 four-choice items with options 

prlnte5"in French, based on spoken (recorded) texts of varying length. 
Administration time: about 25 minutes. 

Reading: 45 multiple-choice items, options printed in French, based 

on short sentences or paragraphs. Suggested administration time. 

35 minutes . 

Writing: 28 items of the "fill-in" type, where a word had to be sup- 

plle3 from the context. 

5 items requiring the u correct French reply 11 to questions 9 usually 
involving change of person, use of the negative, use of pronouns, etc. 

8 items involving rewriting sentences to change number, person, etc. 

1 item involving writing of a 6-line "directed" dialogue. 

Suggested administration time: 35 minutes. 

Sneaking: Test S2. Individual administration, about 15 minutes per 

pupil o 

Four parts: . 

Pronunciation : (Same as corresponding section of Test Sl,j 

Structural Control : Similar to the corresponding section of Test SI, 

but~with”different "quest ions and pictures. 

Oral Reading: Pupil reads aloud a French paragraph. 

Fluencg 1: Pupil chooses one of three picture-sequencies, to recount 

the”story"Iraplied by the pictures. 

Fluency 2: Pupil chooses one of three pictures showing certain situa- 

tions ;"he"Is to tell, in French, "what has happened," "what is happen- 
ing now," and "what will happen next." 

Population IV: (Survey tests) 

Listening: 12 four-picture items 

50 four-choice items with options printed in French. 

Administration time: about 30 minutes. 

Rssdiiig- 53 four— choice items, options printed in French, based on 
sEort passages in French. Suggested administration time* 40 minutes. 
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Writing J (There was no special writing test for this population; where 
one~was administered, it was usually that for Population IVS.) 

(^sre was no special Speaking Test for this population; 
where'one was administered, it was that for Population II.) 

£j2£22i£5 s Tes * 5 four-picture items, jilus 46 four-choice items 

w!th“options printed in French. Administration time: 30 minutes. 

goading : 44 four-choice items based on passages, dialogues, etc. of 

Varying lengths. Suggested administration time: 33 minutes. 

Writing : 53 ’’fill-in" items; 7 "correct reply" items; 7 items requir- 
ing grammatical changes in sentences; 2 "directed essays": student 

was to write an essay using certain suggested words and phrases. Sug- 
gested administration time: 35 minutes. 

The revised definitions of target populations are as follows: 

£222i2£i2*L l! A11 students aged 10.0 - 10.11 at the end of the school 

year who will at that time have completed two or more school years of 
instruction in French (whether or not they are still studying it). 

P2gulat i 2 n_II i All students aged 14.0 - 14.11 at the end of the school 
year who will at that time have completed two or more school years of 
instruction in French (whether or not they are still studying it). 

Pogulation_III: All students at that point of secondary schooling where 

tEere Is a major drop-out from French (or from school altogether) who 
will at the end of the school year have completed two or more school 
years of instruction in French (whether or not they are still studying it. 

P 2 £ulation_IV: All students in the pre-university year (approximate grade 

level: Qracie XII) who at the end of the school year will have been study- 

ing French for two or more years (whether they are still studying it or 
not). 

^2i?2i2£*2£_5[5 : AH students in the pre-university year who have studied 

French for at least three years and are still studying it. (National 
Centers are to make operational definitions of this target population for 
their own countries. These students are those, in the country, who are 
regarded as specializing in French. The operational definitions are to 
be sent to the IEA coordinator.) 

An indication of the quantity of pretesting is given here: 
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Population I tests: 



Listening LI 

England (N=135) 
U.S.A. (Nsl02) 
Scotland (N=28) 

Listening L2 

England (N=150) 
U,S.A. (N=138) 



Speaking* 



England 

U.S.A, 



Reading HI 

England (N=152) 

U.S.A, (N=102) 

Scotland (N=29) 

Reading R2 

England (N=138) 

U.S.A. (N=137) 

(Note: Also, Listening LI, L2 and Reading Rl, R2 were given a sample 

of 160 French children, 80 children took tests Ll and Rl; 80 took tests 
L2 and R2. This was to test for possible ambiguities in the items.) 



Population XI tests: 



Listening 


Reading Speaking* 


Writing* 


England (N=120) 
Iran (N=9l) 
Scotland (N=l49) 
U.S.A. (N=155> 


England (N=117) England 
Iran (N=102) Scotland 
Scotland(N=l49)U.S.A. 
U.S.A. (N=154) 


England 

Iran 

Scotland 

U.S.A. 


Population IV tests: 




Listening 


Reading Speaking* 


Writing* 


Chile (N=159) 
England (N=;119) 
Iran (N=1Q0) 
Scotland (N=l44) 
Sweden (N=l4l) 
U.S.A. (N=95) 


Chile (N=l69) Chile 
]&igland (N=126) England 
Iran (N=99) Scotland 

Scotland (N=150)U.S.A. 
Sweden (N=13l) 

U.S.A. (N=l47) 


England 

Scotland 

U.S.A. 



Population IVS tests: 

Listening Reading 

Chile (N=59) 

England (N=77) 

Scotland (N=88) 

U.S.A. (N=220) 



Speaking 



Chile (N=80) 
England (N=70) 
Scotland (N=93) 
U.S.A. (N=2l8) 



Writing* 

Chile 

England 

Scotland 

U.S.A. 



The objective tests were item— analyzed at the Nationax Centers 
according to instructions that had been sent out by IEA. The Speaking 



♦Tests results for Speaking and Writing wei^ received for only small 
samples of the tested populations. / 



and Writing tests were scored at the National Centers * but item-analysis 
procedures were not applied. Instead, samples of test responses (usually, 
a one-tenth sample of the tested population) were forwarded to Hamburg 
for evaluation by the International Committee* 

A meeting of the International Committee was held in November 25 - 
29, 1968 to consider the item analysis results and prepare semi-final 
forms of the tests. Although many of the test items were found to be 
effective in terms of difficulty and discrimination indices, many others 
had to be revised or rejected altogether. Nevertheless , it was possible 
to develop semi-final forms of the Listening and Reading tests for all 
populations. On the basis of the limited results available for Speaking 
and Writing tests, it was decided that fairly extensive revisions for 
these tests were needed, necessitating the prospect of another round of 
pre-testing to be done in the spring of 1969* In fact, it was thought 
desirable to have another round of pre-testing for all the tests in view 
of the revisions that had been made* 



5.4 Preparation of Penultimate Versions of Tests 

The semi-final forms of tests constructed at this meeting may be 
described as follows t 



Population I 
Listening: 

Reading: 



40 four-picture format items; 20 minutes working time plus 
administration time, stimuli on tape* 

40 four-picture format items, stimuli printed in French; 

25 minutes working time, plus administration time. 



Speaking: Three parts, as before t 

Revised pronunciation test 
Revised structural control test 

Fluency: revised marking scale and instructions for its 

use. Revised pictorial material* 



Ijpulation II 

Listening: 40 items (some of the four-picture type; others with op- 

tions printed in French) ; 20 minutes plus administration 

time* 



Reading: 

Speaking: 



Writing: 



40 items; 25 minutes plus administration time* 

Pronunciation (revised) 

Structural control 

Oral reading (shortened paragraph) 

Fluency: Revised marking scale. 

Total administration time: about 15 minutes* 

Section 1; 2? one word completion items. 

Section 2; 8 multiple completion items (20 points) 
Section 3; free composition passage* 

Total administration time: about 35 minutes* 
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Population XV 

Listening i 45 items; 25 minutes , plus administration time. 



Heading! 40 items; 25 minutes, plus administration time. 

Speaking: (same test as for Population IX) 

Writing! Section l! 16 one word completion items. 

Section 2: 12 multiple completion items (57 points). 

Section 3 : free composition, two passages. 

(a few additional items will also be included in Sections 1 
and 2). Total administration time! about 40 minutes. 



Population 1VS. 

Listening: 45 items; 25 minutes, plus administration time. 

Beading: 45 items; 30 minutes, plus administration time. 

Speaking: (Same as for Population XX.) 

Writing: (Same as for Population XV.) 



•\ Listening to the Speaking test tapes that had been sent in and 
comparing the responses to the marks that had been assigned locally, 
the members of the committee judged that the marking was too variable 
across countries to continue a policy of local scoring. Therefore, it 
was decided to set up, in the future, a central scoring facility where 
raters could be trained to more or less uniform marking standards across 
countries. Provision will be made so that each Speaking test sample 
will be marked by three independent raters. The same will apply to 
Section 3 of all Writing tests. 



The tests presented in Volume II, Booklet 5 are the pre-test 
versions » 



5.5 Timetable for Future Activities_of_the_Committee 



Dec. 1968 - Jan. 1969 * 
Feb. - Hay 1969* 

May - June 1969* 

June - July 1969* 



Preparation of second pre-test forms, tapes in- 
structions, etc., suitable for sending out to 
the participating countries. 

National Centers prepare for pre-testing; item 
analyses to be sent to Hamburg by the beginning 
of June. 

IEA Secretariat and Central Scoring Committee 
prepares materials for a summer meeting of the 
International Committee. 

Meeting of the Committee to review second-round 
of item-analysis results, review results on stu- 
dent and teacher questionnaires, attitude: and 
exposure scales, and refine lists of hypotheses 
variables. (Tentative dates and venue of 
meeting: Nadrin, Belgium, June 27th - 4th July) 



July — Dec* 1969* 



staff prepares materials for ’dummy-run 
lg f in P Spring 1970 (answer cards, instruc- 



CHAPTER 6 



ENGLISH AS A FOREIGN LANGUAGE 



6*0 Introduction 

The original members of the International Committee for English 
which first met by invitation of I.E.A. at Hamburg in November 1966 were 

E. Glyn Lewis (Chairman) H.M.I. and later Director of Educational 
Research at University of Wales, Swansea 

Mrs. Heitner, Israel Department of Education 

Monsieur Morette, Senior Master at the Lycee Henri IV, Paris 

Dr. Agnes Niyekawa-Howard , University of Hawaii 

•• • • 

Mr. R. Piirtola, University of Jyvaskyla, Finland 

Professor C. H. Prator, U.C.L.A., United States of America 

Subsequently, Mr. Ian Dunlop of the British Centre, Stockholm was in- 
vited to join the Committee. He has participated in the work of the 
Committee by correspondence from the beginning. 

6.1 Preparation of Rationale 

The Chairman presented the Committee with a paper outlining pro- 
visionally the variables which might have to be taken into account in 
designing the study in line with the intentions of I.E.A. , and this was 
accepted with some modification at the first meeting. In the light of 
the briefing the Committee had received, it set out a series of 24 hypo- 
theses which could form the guidelines for the study, and indicate at 
the same time the kinds of research instruments tests and descriptive 
measures which would have to be designed. It was agreed that only two 
major target populations should be investigated - I.E.A. Populations II 
(and III), and IV. It was decided that Population I, namely the 10- 
year-olds would not provide a sufficiently rewarding and satisfactory 
sample in the countries likely to participate in the study, since too 
few children would have been taught English long enough to justify test- 
ing at that age. 

The Committee prepared a provisional 'content analysis' formula 
to bo submitted for completion by the National Centers, and a second 
formula for completion relating to the objectives of English teaching 
in each of the participating countries. The areas of skill and knowledge 




to be studied were also discussed, and it was agreed that no attempt 
would be made to test knowledge of the history and cultural background 
of English speaking countries, and that the achievement tests would be 
directed solely to investigating the control over the language possessed 
by the students# It was agreed also that the tests of proficiency should 
take into account as wide a range as possible of actual achievement in 
view of the wide currency of the English language and the fact that in 
some countries it was the first foreign language and taught to a high 
proportion of the school population. Finally at the first meeting some 
consideration was given to the form the tests should take# It was agreed 
that of the two alternative approaches to the preparation of these tests - 
namely, first to design the tests according to the linguistic level rele- 
vant at that point such as vocabulary, grammar or the sound system of the 
language; or second, the language skills possessed by the student namely 
listening, reading, speaking and writing, the Committee would adopt a 
consistent approach and follow the second criterion and so prepare lis- 
tening, reading, writing and speaking tests ensuring at the same time 
that the important linguistic aspects were satisfactorily covered ac- 
cording to the content analysis agreed upon by the Committee. It should 
be emphasized that several of the initial decisions have had to be re- 
viewed (and in some cases revised) by the International Committee in the 
light of the views expressed by National Centers and the experience of 
attempting to implement those decisions. For this reason a section of 
this report is devoted to a consideration of some of the more interest- 
ing issues that have had to be resolved. 

Following the first meeting, the National Centers were informed 
about the preliminary thinking of the International Committee. They were 
invited to comment upon the approach suggested and to supply information 
concerning the content analysis of the curriculum and the teaching ob- 
jectives which they entertained. By and large the response of the 
National Centers proved encouraging, although one or two questioned t he 
possibility of creating uniform international tests, and the advisability 
of using multiple choice techniques. 

6.2 Preparation of Materials - First Stage 

The second meeting of the International Committee considered the 
replies from National Centers and decided on the proportion of testing 
time to be devoted to the various skills. It was agreed that the greater 
proportion of time should be given to reading and writing items, and 
that speaking tests should be restricted to a sub-sample of the sample 
of the relevant target populations. The International Committee also 
considered a paper which collated the content analyses submitted by the 
Natio nal Centers, and a second paper which collated their statements of 
teaching objectives. The Committee modified both papers slightly and 
then accepted a collated content analysis on which the preparation of 
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the actual test items would be based* They accepted also a formula of 
teaching objectives which was to provide a basis for the proportionate 
emphasis to be placed in each target population upon the aspects of the 
English language to be tested. This information was conveyed to the 
National Centers who were also informed that 

Listening_Tests would be multiple choice tests involving oral 

written responses. The whole test would be in 
English. 

Reading Tests would include items of varying length covering 

all aspects of the language - vocabulary, struc- 
ture, etc., with multiple choice options in 
English exclusively. 

Writing would be tested incidentally by means of the 

reading tests and the written responses to the 
listening tests. 

The same considerations would apply to Population II as to Population IV, 
save that in the latter case passages of greater length and complexity 
would be introduced into the Reading tests. It was decided to delay con- 
sideration of the form of the speaking tests until greater information 
had been obtained from I.E.A. concerning the nature of the sub-sample 
to be tested. The preparation of these tests was left to the Chairman 
in the first instance.. It was also decided that there should be a con- 
siderable overlap between the two major target populations so far as 
test items was concerned, and that this would meet the needs of the 
"Survey" test component of Population IV, At the same time it was agreed 
that a considerable part of the Population IV tests should be of a quite 
advanced standard. With all this in mind the National Centers were asked 
to supply as large a selection of test items thought suitable in their 
countries from which a selection of the first draft test batteries could 
be prepared. 

The National Centers responded with useful comments and very 
comprehensive selections of test items for the relevant target popula- 
tions. The Chairman then prepared a provisional battery of tests for 
each population based upon the selection offered by National Centers, 
published and non-res trie ted materials in Britain, and material sup- 
plied from non-restricted items in the possession of the English Language 
Center at Ann Arbor, Michigan and the English Language Center at George- 
town University. These batteries were then submitted to the National 
Centers and members of the International Committee for comment. They 
were returned with useful corrections and comment and the Chairman re- 
vised them in the light of these comments. Finally they were discussed 










wm 



mmm. 






HU! 






and refined by the Chairman in consultation with Mr. Piirtola at a series 
of sessions at the National Foundation for Educational Research, London. 
Following the final revision, the tests were submitted to two groups of 
thirty children of appropriate ages in schools in Wales where the English 
language is not the mother tongue of the majority of .the children. This 
was done to test for gross ambiguities and unsuspected errors or diffi- 
culties. Very few of these were discovered. Consequently the test items 
were made up into 8 units of testing material for Population II and 10 
units for Population IV. Each unit consisted of no less than three test- 
items and not more than ^5. Test items were repeated in some instances 
in several units so that a comparison between the response of various 
pre-testing contres might be established. The pre-test units were then 
submitted to various National Centers for pre-testing. 

Breakdown of Pre-tests 



The following is a breakdown of the composition of the tests 
prepared for pre-testing - 

t 

Population II 



Listening 



i Dictation. 

ii Phonemic identification tested by picture and multiple choice op- 
tion of taped words. 

iii Listening comprehension tested by taped short statement or series 
of statements and multiple choice response. 

iv Recognition of sounds - differences and identity - by means of a 
series of three taped words. 

v Recognition of differences or identity of intonation in three taped 
sentences identical in structure but sometimes different in intona- 
tional pattern. 

Reading 

i Four different types of tests relating to pronunciation, spelling 
and stress. 

ii Eleven tests of structure control including the use of some tests 
involving pictures. These tests cover all the aspects included in 
the content analysis agreed upon, including especially word order 
and the ability to transpose direct and indirect speech, active and 
passive, question and statement. 

iii Vocabulary - eleven different types of tests some of these being 
incidental to listening and reading, but some specifically designed 
for vocabulary alone - namely tests of definition, synonyms and 
antonyms in and without context. 
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iv Recognition of meaning was tested partly by the identification of 
narrative or descriptive detail in continuous passages or by means 
of the recognition of correct inference or incidentally in the 
tests listed above. 



Population IV 



Listening 



i Dictation. 

ii Recognition of intonation patterns combined with differences of 
emphasis. 

iii Listening comprehension as Population II - iii. 

iv Listening comprehension tested by taped short conversations and 
with choice of three written responses. 

v As for Po pulr t ion II - iv. 

vi Listening comprehension involving recognition of key phonemes for 
discrimination of meaning-. 

vii As for Population II - v. 

viii Listening comprehension based on choice of correct response to 
fairly simple taped statements. 

ix Identification of correct stimulus sentence given the correct writ- 
ten response. The identification from among three stimulus sen- 
tences depends upon the correct relation of intonation and emphasis 
to meaning. 

x Straightforward listening comprehension of details of description 
and narrative. 

Reading 

i Tests on recognition of correct stress related also to the differ- 
entiation of nouns and verbs. 

ii Grammar and structure - These are very largely as for Population II 
in form though not in content. More emphasis on transform tests, 
and on the use of extended passages for the recognition of details 
of correct usage. These tests are basically miscellaneous in 
character; they do not deal with categories of structure and usage 
in blocks of items. Nevertheless each test item is easily iden- 
tifiable for analysis in terms of the element it tests. 

iii Vocabulary - Very largely the same in form as for Population II. 

iv Style - Tests have been included which seek to investigate perform- 
ance in recognition of appropriate registers and styles. These 
tests take the form of linking disparate statements to form a co- 
herent passage or paragraph. For each possible statement, a mul- 
tiple choice of styles or registers is offered. 

v Comprehension - Several tests of comprehension are provided - some 
devoted to the ability to draw correct inferences, to come to cor- 
rect conclusions, to recognise possible implications, as well as 
straightforward recognition of descriptive or narrative detail. 
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Speaking and Writing Tests 

i Continuous oral composition subsequent to provision of eight 
stimulus items. 

ii Oral description and narrative, based on series of pictures. 

iii Mimicry-repetition of spoken sentences. 

iv Oral response to specific taped questions referring to taped 
short statements. 

v Oral response to specific taped questions referring to taped 
short statements. 

vi Single word taped response to simple pictures of single objects 
or actions. 

vii Written completion of sentences. This test is to some extent 
open-ended since what is required is a correct response rather 
than the only correct response possible. 

viii Continuous written composition, including letters, based upon 
’’guided composition principle." 

6.3 Pretesting 

The following pre-tests were administered: 



Population II: 



Listening 




Reading 




Germany 


(Test 11) 


Germany 


(Tests 1,2, 3, 4, 5, 8) 


Finland 


(Test 12) 


Finland 


(Tests 3,7,8,9,10) 


Iran 


(Test 11) 


Iran 


(Tests 1,3,6,7,9,10) 


Italy 


(Test 12) 


Italy 


(Tests 2,4,6,8,10) 


Netherlands 


(Test 11) 


Netherlands 


(Tests 1,3, 4, 5, 7, 9) 


Sweden 


(Tests 11 $ 12) 


Sweden 


(Tests 2,4,6,8,10) 


Thailand 


(Test 12) 


Thailand 


(Tests 1,2, 3, 3, 7, 9) 


Population IV: 








Listening 




Reading 




Chile 


(Tests 9,10) 


Chile 


(Tests 1,2, 3,4) 


Germany 


(Tests 10,11) 


Germany 


(Tests 4, 3, 6, 7) 


Finland 


(Tests 11,12) 


Finland 


(Tests 1,2, 7, 8) 


Iran 


(Tests 9*12) 


Iran 


(Tests 3, 3, 6, 8) 


Italy 


(Tests 10,12) 


Italy 


(Tests 1,2, 3, 4) 


Netherlands 


(Tests 9,11) 


Netherlands 


(Tests 4, 3, 6, 7) 


Sweden 


(Tests 9»10) 


Sweden 


(Tests 3, 5, 6, 8) 


Thailand 


(Tests 11,12) 


Thailand 


(Tests 1,2,7, 8) 



6.4 Attitude and Other Descriptive Measures 

Meanwhile the International Committee with guidance and assist- 
ance from the officers of I.E.A. gave attention to the preparation of 
attitude scales and descriptive measures. Among the descriptive measures 
the English Committee had decided upon are the following: 

i A method analysis. 

ii A measure of linguistic distance. 

iii A description and measure of linguistic background of students. 

iv A description of the linguistic situation in each participating 
country. 

The meetings of the International Committee as a whole did not 
provide the time necessary to develop these instruments. Consequently 
the Chairman took advantage of a visit to the United States for other 
reasons to consult with Dr. Niyekawa-Howard on the preparation of these 
measures. After two days of intensive consultation tentative proposals 
were made to the officers of I.E.A. These were further refined in dis- 
cussion between the Chairman and officers in London and in collaboration 
with Mrs. Burstall of the French International Committee at the Stockholm 
meeting of I.E.A. in December, 1967* 



6.5 Points of Principle Raised During the Study 
and Justification of the Committee's Approach 

(a) An instrument for the analysis of method 

The English Committee has no desire to exaggerate the importance 
of difference of method in helping to determine levels of achievement. 
Nevertheless the Committee has always wished to make some attempt at 
such an analysis, and to relate it to other variables. The only ques- 
tion to undertake this specifically for language is as follows: 

Indicate by checking in the appropriate column the amount of 
importance in general of each of the following in learning 
English/French as a foreign language. 
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Important Irap0rtant Important 



(a) Listening 



(b) Speaking Correctly 



(c) Speaking Fluently 



(d) Understanding Spoken 
English/French 



(e) Use of Formal Grammatical 
Drills 



(f) Grammatical Explanations 



(g) Beading for Clear 
Understanding 



(h) Reading Extensively 



Cl) Writing Correctly 



(j) Writing Fluently 



We wish to include other questions which would relate to the teaching 
of language specifically, since we are convinced that questions relating 
to teacher-pupil relationship and to teaching method in general terms 
are more than likely to miss the relevant and significant aspects we 
wish to analyse. 



(b) Measure of language distance 



The problem of negative or positive transfer from the act of 
learning one language to the act of learning pother is important, simply 
because the transfer will be facilitated not so much by the qualities 
possessed by the student or by the teacher, or the system under which 
the language is taught as by the relationship which may exist between 
the two languages - the distance -of the two languages. This operates 
at all levels, vocabulary, the sound system and the grammar. One way 
of taking this factor into account would be to use a contrastive analysis 
of the mother tongue and English. But apart from the fact that this 
process would be enormously expensive in terms of time and money and 
would hold up the preparation of the achievement tests for a considerable 
time, it would not solve the problem of preparing a set of tests which'' 
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have to be administered across the board - a set of tests to be ap- 
plied to all cou n tries in common* A contrastive analysis wouJ.d enable 
us to prepare tests which were fair to each country separately since 
they would take each native language into account. But it would not 
enable us to provide the basis for comparison of achievement. 

The procedure adopted by the English Committee is not entirely or 
even broadly fair to all countries, but if we can include a measure of 
distance, it can be said to be equally unfair and the differences in 
unfairness can be partialled out. The procedure is to ask the teachers 
first of all whether they speak another foreign language and if they do 
to indicate whether the learning of the two languages, in their view 
is equally difficult or more/less difficult. We ought to get a compari- 
son of the difficulty of learning English expressed in terms of the dif- 
ficulty of learning several other foreign languages, and this informa- 
tion will enable obtaining a measure of the distance of English from 
each of these languages. It is accepted that this measure is subjective, 
but it will be based on the best judgment of experts. But in order to 
refine this rough instrument a little we would like to include the ques- 
tions which refer to this issue in the questions which may be addressed 
to National Centers with a request that the National English Committee 
in each case should answer it. 

A second approach to^the assessment of linguistic distance is in 
the Teacher Questionnaire. This is an attempt to identify fairly pre- 
cisely the half dozen/or so most important areas of linguistic inter- 
ference, or differences between the mother tongue and English. We wish 
to correlate in/the best way possible the results from this question 
and the error/analysis of the achievement tests in their final form. 

Two results/would be obtained in this ways a check on the a priori 
theoretical contrastive analysis given by the teachers in reply to the 
relevant question. And second, in so far as the correlation was positive 
an (^significant , it would be possible to explain the incidence of errors 

which occur. 

(c) Which English? 

The v ar io us National Committees were asked to indicate whether 
they preferred the British or American variety of English where the dif- 
ferences between the two might be important. This was done because some 
of the participating countries are likely to be under the influence of 
one rather than the other. In the event the National Committees who 
were addressed decided in favour of British English. Since that time 
one or two other countries have expressed a wish to participate, e.g.,. 
Thailand, and it may, be that such countries might prefer American English. 
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(d) Selection of vocabulary and structures 



English is not favourably placed when it comes to decide upon the 
criteria of selection to be adopted in an international test. A good 
deal of very good work has been done in this area in the U.S. and Britain 
but simply because, unlike French, there has never been an official sup- 
port for one set of criteria or for a course employing a stated set of 
criteria (as in Francais Fondaraental) , the teaching of English as a 
foreign language in different countries reflects the acceptance of 
several different criteria. If we adopted one set of criteria coun- 
tries using it would be advantaged compared with countries who based 
their courses on a different set. Consequently we decided to work 
pragmatically and empirically, using the broad experience of the Inter- 
national Committee, corrected by the feed-back from National Centers. 

This procedure again is not fair to any country, but it has the advan- 
tage of being equally unfair to all. 

(e) Grading of vocabulary and structures 

The steepness of the grading of the tests has been a considerable 
problem since, to be satisfactory, every topic tested had to be repre- 
sented several times and this makes for a lengthy test. If in addition 
we decided on a too gradual grading, the tests would become impossibly 
long. It was decided to have a very lengthy set of pre-tests allowing 
for repetition of types, and gradual grading. The final tests will be 
within the time limits specified by I.E.A. and within these limits the 
grading will be as gradual as possible. 

(f) Multiple choice? 

The English Committee decided on this form of testing without any 
considerable discussion, partly because it has proved a successful method 
of setting out language tests in Britain and the U.S. aimed at foreign 
students of English; and partly because the nature of the present exer- 
cise and the resources available made this form of test very convenient. 
Nevertheless one National Center has expressed its reservations about the 
appropriateness and even validity of this kind of testing when applied 
to English. 

(g) Reflection of differences in national objectives for English 

When National Centers were asked to state the objectives pursued 
by their own teachers of English, there were some considerable diver- 
gences, and it has been a problem to try to include sufficient tests at 
each of the target population levels to meet the varying aims. This 
matter mainly affects the balance of the tests - the relative weight to 
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be riven to the varying skills which are tested, some countries favour- 
ing^reading as the ma^ objective, while others fayouredspeechOthers 
a^Iin favoured translation as a major objective while others discounted 

it entirely. 



(h) Use of the mother tongue in testing 



Apart from the instructions, the mother tongue, it was decided, 
should not be used in these tests. This decision effectively ruled out 
certain types of tests, especially translation of passages or the inter- 
pretation of vocabulary in terms of the vocabulary of the native languag . 



(i) Cultural bias 



Just as there are linguistic problems of positive and negative (| 
transfer, so there are cultural problems, or issues of culture ^stance. 
Generally speaking Western European countries are nearer so far as the 
oattern of their culture goes, to that which the English language is 
normallyrelated to, thanis the case with Middle Eastern or Far Eastern 
or Asian countries. Consequently the English Committee hadtodecide 
on one of two alternatives. First, to relate toe content of the 
to the pattern of an English speaking setting. An assumption underlying 
the choice of this alternative is that all countries participating in 
this exercise taught English with the objective of participating^^ 
English speaking culture or at least of entering into an empathic under- 
standing cf it. This is not a valid assumption since some countries 
testing^English are concerned almost exclusively with English as a ve- 
hicular language rather than as a means of culture transmission. The 
second alternative is to relate the content of the features of °^inary 
communication. Since this is basic to all teaching of English, 

Committee decided in favour of the second alternative. It does m ®J n 
the other hand that some countries, e.g., Sweden which places great store 
on their relating the language to the English "way of life 1 (and this 
is becoming increasingly a feature of progressive teaching) are not 
being given the true assessment of their approach to and teaching of 

English. 



Apart from the problem of culture content as it affects different 
countries, there is the similar problem as it affects schools within 
the same educational systems but with different educational orientation. 
This is particularly the case when we consider vocational as opposed to 
academic schools. Subjects which might be within the range of coding 
interest of vocational schools would be outside the range of the academic 
school and the converse is equally true. This has been pointed out by 
one national committee (Italy) when submitting observations on the choice 
of reading passages. In the last resort, unless the list of reading 
passages is to be extended unduly, the choice of cultural content has 
to be left to an arbitrary decision, though we hope it is an informed on 



6-11 



I 



I 






1 












I 



i 



























1 



i 



A 



(j) Cognitive aspect of the language tests 



It has been accepted all along that any tests of linguistic com- 
petence, beyond the most elementary, and excluding the purely imitative, 
are at the same time an assessment of cognitive processes. Consequently, 
especially in the tests for Population IV, but to some extent also for 
other populations, there are several subtests which are tests of the 
ability to draw conclusions, to infer, and to draw out the implications 
of the statements made in English. 

6.6 Future Action 



There have been delays for various reasons, in the pre-testing of 
the English items. It is expected that all item-analysis will be com- 
plete for the next meeting of the English committee in June/July 1969* 
The tests presented in Volume II, Booklet 5 are the pre-test versions. 
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CHAPTER 7 



CIVIC EDUCATION 



7 . 0 Introduction : 

The assessment of children* s achievement in Civic Education pre- 
sents a number of basic problems which highlight the differences between 
Civic Education and other school subjects. 

One of these is the fact that Civic Education is not yet well- 
established as an academic discipline, so that there is a good deal of 
disparity between countries in the subject-matter covered and the ways 
in which it is approached. Moreover, these disparities are more in- 
fluenced by a particular country’s political and socio-economic system 
than is the case, say, with Science or Mathematics. Thus, in a Monarchy 
children are likely to learn a good deal more about royal succession and 
duties to the Crown than in a Republic. 

Another difference lies in the fact — much stressed both in the 
curricular objectives and by the teachers themselves — that Civic Edu- 
cation does not merely consist in the transmission of a body of knowledge , 
but that it aims at inculcating certain common a ttitudes and values, such 
as a liberal and democratic outlook, political responsibility, the ideals 
of tolerance and social justice, respect for authority, and so on. Be- 
hind the need to teach pupils certain fundamentals about political, 
economic and social activities and organisations lies the desire to turn 
them into effective and responsible citizens of their society. Indeed 
the cognitive content of the curriculum is frequently used in order to 
highlight the underlying principles and ideology; thus, information about 
electoral systems could be utilised to bring out fundamental ideas about 
equality and majority rule. 

One implication of this has been that, as we shall see, the measure- 
ment of the children’s knowledge (cognitive domain) and the measurement 
of their attitudes and values (affective domain) has proceeded somewhat 
separately — partly because children’s attitudes in this sphere so often 
develop well before they have the relevant knowledge, and partly because 
affective measures need a rather different type of instrument (attitude 
scales, projective techniques, and so forth). At a later stage the data 
obtained through these two sets of measures will be compared and con- 
trasted, and given a theoretical interpretation. 

A third difference between Civic Education and other subjects lies 
in the very considerable influence which out-of-school factors will have 
on the pupil’s knowledge and attitudes. We must expect that the child 
will be affected not only by what he learns in the classroom, but also 
by political events and by the entire fabric of the society in which he 
lives. As far as possible we must, therefore, make an effort to include 
measures of these among our independent variables. 
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Finally, we must mention a linguistic and terminological difficulty: 
in writing about this area, it is difficult to avoid over-simplified la- 
bels such as *patriotism, * •democracy, 1 * equality of opportunity,* * citi- 
zenship* and the like, but we should like it to be clearly understood 
that Guch terms are used by us purely for reasons of convenience and not 
in order to advance any ideological point of view. The labels merely 
refer to particular curriculum segments or to variables in the conceptual 
framework; they are defined by their content specifications. 



In short, we find that the assessment of achievement in Civic Edu- 
cation cross-na tionally is attended by a number of awkward problems: un- 



certainty about a * common core,* the need for special attitude measures 



as well as knowledge tests, strong out-of -school influences, terminology 
with political overtones creating difficulties in conceptualisation. We 
have addressed ourselves as best we could to each of these, as will be 
shown in the succeeding sections. 



7.1 Contents of the NationalDocuments:. 



The members of the International Civic Education Committee of I,E,A, 
which met in Home in April, 1967* had before them a series of documents 
produced by the national centres of W, Germany, Finland, Iran, Sweden, 
the UK and the USA, These documents contained a digest, for each country, 
of the curricular aims and contents in Civic Education for different age 
groups, of textbooks used, and of examination contents. While the coun- 
try documents differed widely, it was obvious from the start that, in 
addition to knowledge, all countries laid considerable emphasis on values 
and attitudes. Here, for instance, is section (l) of article 26 of the 
constitution of the German state of Bremen: 



"Children should be brought up in a community spirit based on re- 
spect for the dignity of all men and on desire for social justice 
and political responsibility, and should be taught to regard the 
opinions of others with objectiveness and toleration and work 
peacefully with other people and nations,** 



We therefore undertook two detailed content analyses, one dealing with 
cognitive knowledge, the other with values and other affective variables. 



The content analysis of the cognitive domain led in due course to 
a conceptual framework (see Appendix H-l) which has been the major guid- 
ing force in the development of cognitive measuring instruments. The 
analysis of the affective contents of the country documents showed that, 
perhaps a little unexpectedly, there was a common core of basic values 
which are stressed in all six countries concerned. This common core 
consisted of the political ideology of equality (freedom and civil liber- 
ties, tolerance of diversity, majority rule, non-violent conflict reso- 
lution, etc.) and citizenship values (such as informed participation, 
being law-abiding, interest in welfare of fellow citizens, love of country) 
These values were set out in detail, and were in due course incorporated 
in the conceptual framework and in the measuring instruments* 
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, 2 The Development of Cognitive Instruments 



Between November, 1966 and September, 1967 the international civic 
education committee of the IEA study formulated its preliminary test 
specifications. The first meeting produced a preliminary rationale and 
model stressing structures of authority. This model was used as an 
overall conceptual framework which would guide the inquiry until in- 
formation could be gathered about civic education from documents which 
national centers in the seven participating nations would produce. 
National centers were asked to prepare papers which would summarize 
the major goals of civic education in their country, representative 
curricular statements, examples of test materials in use, and summaries 
of textbook materials used in civics or other social studies courses 
ha ving political content. These reports were received in the spring 
of 1967 and provided the basis for a subsequent meeting. 

Before and during this meeting a content analysis of the national 
documents also was used to formulate an international master grid (test 
content specifications) summarizing the major content and ability areas 
of civic education as taught in the seven countries at the three basic 
target population levels! the 10 year olds, 14 year olds, and pre- 
university year students. Bach of the cells in these test specifica- 
tions was given a weight showing the relative emphasis expressed in the 
natural documents received from the seven countries. Test questions 
received from the various national centers were then categorized in 
terms of this master grid. These specifications with the illustrative 
questions went out for review to the various national centers with a 
request that new questions be written in accordance with the specifica- 
tions. When the reviews and questions were received this then provided 
the committee with the necessary material to produce definite test 
specifications at a meeting in September, 1967* 

7 . 3 The Civic Fducation^Content_Specifications 

The results of our review of national center reactions to the 
preliminary test specifications indicated that terms such as civics, 
citizenship, or political education were meaningful educational terms 
( as courses of study or curricula emphasis) in all seven countries. 
However, these terms had varying degrees of currency in the educational 
languages of participating nations. In some countries "civics” is 
treated as a separate subject. In others it is taught through history 
classes while in still others it is a form of school work emphasized 
throughout the curriculum. For instance in Sweden, which has formal 



courses in "civics," topics such as working life and economic problems, 
international problems, the political process (e.g., public opinion and 
political parties), law and justice, personal rights and duties, and so 
forth are taught in "civics" courses. In the United Kingdom, which has 
little formal training in "civics," these same topics are taught through 
several subjects. 

Our content analysis and survey results revealed that the common 
core of civics taught in the seven nations dealt with the following 
major topics: 

I. Fhndhmentals_and_Nature_of_Citizenshig - citizenship concepts and 
definItIons - of~terms - such”asT state7~government , patriotism, freedom 
and responsibilities, rule of law and democracy; 

II. Political_Processes_and_Institutions t National Topics such as con- 
si?itutIons7”poIItIcaI”EIstory7”IocaI~government, law making, judicial 
decision making, executive policy making and the civil service, political 
leadership, public opinion, elections and political parties; 

III. PoliticalJProcesses_and_Institutions: Intemational_Topics such 

as foreIgn - poIIcy7”na : EIonaI~3e?ense7 - coipara£Ive”poII£ics7~the~Inter- 
national political system, and interna tional organizations; 

IV. Economic_Processes_and_Institutions - topics such as social ser- 
vices7~weI?are7”taxation7”Iabor~organizations, and occupations; 

V. Social Processes_and Institutions - topics such as the family, the 
schooI7”masi - me3Ia7”traffic7 - and"crIme. 

Each of these topics was also divided in terms of simple, complex, 
and abstract operations a la Bloom's taxonomy. In all, approximately 
1,300 draft pretest items were submitted to the international committee 
for possible inclusion in the draft pretests. 

7.4 The Civic Education Pretests 



On the basis of these content specifications and the draft items, 
general forms of pretests were prepared for four groups of students, 
namely the 10-year-olds, 14-year-olds , and preuniversity year populations 
and a series of tests for those preuniversity year students who have 
taken extra work in civics, history, and social studies ( a specialist 
group). These pretests (about 850 items) have been or are being ad- 
ministered in each of the participating countries. On the basis of 
these results the final pretest instruments are being prepared. 
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In addition to the test analysis information resulting from the 
pretesting, National Centers have also been providing the civic educa- 
tion committee with feedback information as to whether the content of 
test questions is appropriate for the various age levels and if the 
content is suitable in their countries. This has required them to re- 
vise test questions in some instances so that the tests will be appro- 
priate • 

Two other checks of content validity have been or will be made. 
First a check list of content in the attitudinal and cognitive domain 
has gone out to national centers and the results are being compiled. 
This will aid in the development of the final pretests. What is ex- 
pected to come out of this testing is a series of subscores for each 
of the five (I-V) subtopics of the content specifications described 
above. This will allow us to construct national and international pro* 
files of children* s knowledge of international topics, economic and 
social processes and institutions, citizenship fundamentals, and basic 
national political topics. Certain other and smaller subscores could 
be developed showing comparative knowledge about constitutions and 
political history for example. No national citizenship ’’scores" on 
one civics test are contemplated. Instead a series of national pro- 
files will be produced in which one nation may do well on one sub test, 
poorly on another, and just average on another. This will produce 
more objective international results which will be useful nationally 
and internationally. 

A selection of test questions has been made from the draft pre- 
tests to illustrate the types of questions which will be used to sample 
content in the five major areas of the civic education inquiry. These 
are attached in Appendix H-2. The pre-tests are presented in Volume 
II, Booklet 5. 



Three pretest forms of 35 items each were prepared for population 1, 
three forms of 45 items each for population II, and three forms of 50 
items each for population IVG (General), and five forms of 50 items each 
for population IVS (Specialists). Each of these l4 different forms have 
seven items in common (which sample the content and abilities dimensions 
of the test specifications) so that some estimate of the average ability 
level of the groups being tested can be made both horizontally and ver- 
tically, i.e. ? within the same age groups and among different age groups 
which are being exposed to different forms of the pretests. All pretests 
were designed to be completed within a class period of 45-50 minutes dura- 
tion. The time taken to complete the test as well as the average dif- 
ficulty scores will also give us some indication of the relative diffi- 
culty of the various test forms. 
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7.5 Criteria for Selection of Items for Final Draft Tests 



2 . 



It is necessary to describe the following general criteria which 
will be used in selection of items for the final drafts of the civic 
education cognitive tests for populations I, II, IV, and I VS; 

1* Is this item measuring what we say it should be measuri ng 
i.e. , is it valid? 

Is it important to know this particular piece of information, 
i.e,, on the basis of committee comments, national center 
documents, reviewers* comments, etc.? 

Is the item working well cross nationally, i.e., do the item 
statistics indicate that the item is discriminating well be- 
tween high and low scoring students, is the item within an 
acceptable range of average difficulty in each nation, are 
the dis tractors working, etc. 

These and other such questions will be asked and answered after 
the item analyses from all participating countries are available and 
after the civic education committee has had a chance to review these 
results. By the spring of 1969 this task should be completed and the 
final draft tests available for publication and the dry run by that 
spring. 
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7.6 The Development of Affective Instruments 



If 



As we have seen, the analysis of the content of the country docu- 
ments showed that all the countries concerned emphasized education for 
citizenship and the broad ideological area of equality , egalitarianism 
or democratic values. The International Civics Committee agreed there- 
fore - 

I. to regard these two sets of values as a potential common core 
for cross-national assessment; 



II. to carry out pilot interviews in schools which would lead to 
a more detailed conceptual framework and a better understand- 
ing of childrens developing thought processes in this area; 



III. to proceed somewhat separately from the cognitive domain; 

while recognising the close interlinking of perception, cog- 
nition, values, and behavior tendencies; this was done purely 
for reasons of convenience and does not imply a misleading 



dichotomy. 

Subsequently, the pilot interviews took place in the UK, Sweden, 

W. Germany and Italy. (See Appendix H-3). These were followed by a 
meeting in Apeldoorn in November, 1967, at which a conceptual framework 
was drawn up and agreed to, and at which a certain amount of item writing 
took place. Since then, a number of pilot questionnaires have been 
developed and tried out, in the UK, W. Germany and the USA. The results 
of the analysis of these data will shortly be forthcoming, and will lead 
to fewer and shorter measures with known characteristics in severa 

countries . 



The pilot interviews showed very vividly that political awareness 
and perception of the community develop only gradually. The young child 
tends to focus on himself and the people he knows, and does not perceive 
the community as an organised whole, its services, economy, education 
system, and its many competing interest groups. He trusts adults and 
does not question why or how things are provided, he lacks a sense of 
historical development and future progress, and he cannot conceive of 
adult conflict. His cognitive style is such that he tends to personalise 
and 1 concretise 8 , i.e«, to see political events in familiar personal terms 
rather than as broad principles or institutions: if the British govern- 

ment made a mistake, the landlord of the House of Commons would evict 
them. Everything is seen in simple, stereotyped, good/feiad terms at first, 
with little awareness of finer differentiations or of the possibility of 
criticism and improvement. Politicians, elections, fellow citizens his 
own and other countries are all seen by the child in very simplistic 
terms; only later does he become aware of broader issues such as individual 
freedom vs. the rights of others, civil liberties, majority rule, social 
justice, or the functions of the State. 






o 

ERIC 



7-7 



iL 

























mw 



/ 




/ 









1 




l 

In addition, therefore, to measures of our two broad areas of con- j 

cem (citizenship and egalitarianism), we have developed measures which - j 

hopefully - will serve to demonstrate children’s growing perceptual aware - 
ness of their communities. We, for instance, have asked a number of ques- 
tions which will allow the child to answer in personal or in community- j 

oriented terms, to show his perception of politicians and of elections, j 

his awareness of historical development, or his feelings of closeness/ | 

distance in relation to the national government. Here are some examples: j 

Q. Why do wars sometimes break out? (one answer: "Because some- | 

times countries are too greedy" - tendency to personalise.) 



s 




Q. Why do births, deaths and marriages have to be officially re- 
corded? (one answer: "So that they can know how many births 

toy had and, if too many, try to control it" « less personal- 
ized, more community-oriented.) 

Q # Why does each person have only one vote in an election? (one 
answer: "Because if they had more than one vote they would 
be vot ing forever" - tendency towards concreteness rather than 
seeing the abstract principle.) 

A set of questions asking "How long have we had each of the fol- 
lowing in our country?" — intended to show awareness of progress 
and historical developments (For instance, about a quarter of a 
W. German group of Population II pupils answered, "We have always 
had it" in respect of a King or Queen, women working in factories, 
our political parties, the jury system in courts, advertisements, 
hospitals, and newspapers, while 80# said "We have always had 
voting for men.") 

A set of semantic differential rating scales dealing with the 
child's perception of a. his local town council, b. the national 
government, and c. his local education authority, covering such 
aspects as closeness, friendliness, warmth, understanding, ef- 
ficacy, responsiveness to protest, social justice, power, strict- 
ness (and their opposites). 

The analysis of pre-tests is now proceeding, with a view to improving 
and shortening these measures, and turning open-ended questions into 
closed ones. 

The above examples will by implication have brought out another 
important point: that great care is needed in developing measures of 

values and precepts in children, because these are subtle processes 
which are easily biased by instruments which are too highly structured 
or which impose a pre-conceived, adult framework. For this reason, a 
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variety of techniques has been employed: projective, forced-choice, 

graphic, etc*, and many areas have been covered more than once* Thus 
we hope in due course to develop the methods which are most suitable 
for each age-level* 



The development of the measures for the two main affective areas 
(egalitarianism and citizenship has followed similar stages* First, a 
detailed conceptual framework was drawn up, based on the content analysis 
of the country documents and on the pilot interviews; next, a variety of 
instruments was designed using different techniques (projective ques- 
tions, attitude statements, check lists, forced choice items), which 
have been tried out in W* Germany, the UK and the USA; finally, as the 
result of pre-testing and analysis, a shortened and improved set of 
questionnaires will be finalised for the ' dry run' in 1970* 



The detailed conceptual framework — and hence the measures re- 
lated to it — dealing with egalitarianism or democratic values covers 
principles of equality, civil liberties, majority rule, tolerance of 
diversity, racial equality, etc* Here are some examples of items in 
this field: 



A set of alleged quotations was offered, asking children if they 
might give their support to such a person, or not, and why* For 
example, M 1 think you should stay out of politics," said Mr* C t 
you women have enough to do, looking after the home and bringing 
up the children." Would you give your support to Mr. C? (One 
answer: "No* He thinks that women should not vote and everybody 

over a certain age is allowed to vote and if women want to vote 
they should be left alone." Acceptance of egalitarian principle.) 
Another example: "What is the point of giving a good education to 

boys from poor families?" asked Mr* J, "it will only make them 
want things they can't ever hope to get." Would you support Mr. J? 
(One answer: "No* He thinks that boys from poor families should 

not have an education l 4 ke rich boys families and I think they 
should all have the same education" - Acceptance of egalitarian 
principle* ) 



A set of attitude statements, e.g*, 

The courts of law should give special treatment to people who 
are very rich* 

Most people can be trusted to use their vote in a sensible way* 
Every person should have the same chance as every other person 
to get ahead in the world* 

People of different religions should be allowed to pray in any 
way they like* 
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The analysis will show the factor structure of these measures, the de- 
gree of inter-correlation between them, and the suitability at differ- 
ent age levels. It is perhaps worth mentioning that in these instru- 
ments the emphasis is not on the correctness of the child's knowledge, 
but on his values and attitudes, on his own developing political ideology. 



The conceptual framework regarding citizenship values covered such 
areas as political participation, willingness to serve the community, 
obedience to the law, loyalty, the functions of criticism, interest in 
fellow citizens, interest in other countries, etc. From the pilot inter- 
views we already know that different aspects of the citizen role are 
stressed to children in different countries, and so we have first of all 
designed an inventory dealing with the child's percepts of the components 
and width of the Good Citizen role, in terms of stated ideals. For 
example, the child is asked to indicate whether any of the following 
are included in 'what you mean by a good citizen': A good citizen: 



obeys the law 
is always polite 
loves his parents 
votes in every election 
works hard 

has good table manners 
minds his own business 
is willing to serve on a jury 



and so on (jk items). 



We stress once more that we do not score these items for 'correctness,' 
but for width and emphasis: there may well be a set of 'common core' 

items, which are heavily stressed in all countries, together with varia- 
tions in the degree of inclusion of some of the remaining items, from 
country to country and for children of different ages, intelligence 
levels, backgrounds, etc. In other words, it is a perceptual technique. 



Next, we made several attempts to design a suitable instrument to 
measure the child's own adherence to citizenship values. There are dif- 
ficulties here, since the child is not yet old enough to take the role 
of a citizen, and so we have to ask him what he would do, whom he would 
support, how he would react, and so forth. This inevitably entails ele- 
ments of unreality, and we certainly do not assume that the instruments 
have predictive validity for the child's behaviour as a grown-up; all 
we can hope to find out is what his attitudes are now, in anticipation 
of his role of citizen. To this end, we designed several instruments 
employing different techniques (forced-choice, projective, open-ended 
questions, attitude statements) but covering much the same ground: vo- 

ting, and political participation in general; efficacy, the value of 
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criticism; standing up for citizens* rights; interest in national and 
in foreign affairs; resolution of conflicts: law enforcement; loyalty 
to country; interest in welfare of fellow citizens; etc* Here are some 
examples: 

l "There are millions of other people who will vot? in this election, ** 

said Mr* A, "so it does not matter whether I vote, or not." Will 
you think like Mr. A when you grow up? Yes/No/Don* t Know. 

Why is that? It is because. ... 

Participation: voting in elections. Projective technique.) 

I 

If there were a general election, would you feel 
| (a) that it makes little difference whether you vote, or not? 

OR 

| 

| (b) that you must vote whenever you can? 

(Participation: voting in elections. Forced-choice technique.) 

Lots of little elections are not important enough, to bother about. 

So many people vote in a general election that it would not matter 
whether I voted or not. 

If a person doesn't care about the result of an election, he should 
not vote in it. 

It isn't so important to vote if your party is certain ts lose. 

(Participation: voting in elections. Attitude statement.) 

"If you try hard enough," said Mr. H, "you really can make a change 
in the way the country is run." Will you think like Mr. H when you 
grow up? Yes/No/Don't Know. 

Why is that? It is because. . . • 

(Efficacy. Projective technique.) 

If something is wrong, it does not help to complain about it to 
the authorities. 

(Efficacy. Attitude statement.) 

Our government 

pays attention doesn't pay attention 

to complaints : : : : : . :. to complaints 

can have its can only have its 

decisions changed decisions changed 

by ordinary :::::: by powerful people 
people 

(Efficacy: Semantic Differential rating technique.) 
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We have also designed a confidence index, in which the child is presented 
with a number of minor social problem situations (seeing a house on fire, 
having a noisy neighbour, wanting to meet the local parliamentary repre- 
sentative, borrowing money, getting a marriage licence) which may require 
him to deal with some aspect of authority or community services; in each 
case, we not only ask what he would do, but also whether he 'would feel 
very sure what to do,' 'fairly sure what to do,' or 'uncertain,' As be- 
fore, we are less concerned with the 'correctness' of the answers than 
with the child's self-confidence in his ability to operate the system, 
deal with civil servants, etc. 

Finally, we come to the politicization determinants , the aspects 
of the child's life in and out of school which will need to be measured 
in order to help account for differences in the dependent variables. 

There will be special sets of questions in the Teacher Questionnaire 
but, unfortunately, there can be no Parent Interviews (except, perhaps, 
in the longitudinal study). We have had to restrict ourselves, there- 
fore, to questions which can be answered by the child himself, as part 
of the Student Questionnaire, Here we have the following variables: 

Classroom practices — democratic/authoritarian, (A set of 60 
True/False items abstracted and adopted from the Pace-Stern 
index of institutional characteristics, dealing with partici- 
pation in school decisions, demands for obedience and respect, 
stress on rules, punishment, free discussion, etc.) 

Peergroup practices — how children cope with settling differences 
or changing rules among themselves. 

Mass-media interests in current affairs, etc. (cinema, radio/ 
television, and reading habits). 

Child's interests and activities in current affairs, local and 
national politics (e.g,, help to collect funds, join a protest 
march, make a speech at a mock-election, collect signatures for 
a petition, etc,). 

Frequency of discussion of current events with friends, with par- 
ents, and with teachers, 

Rejection of adult values, identification with the Teen-ager role, 
(40 behaviour items dealing with rebelliousness, precocity and 
disobedience vs, conformity), 

'Peak learning experience' (a set of open-ended questions dealing 

with striking events, persons or experiences which have helped to 

influence the child's attitudes to politics and citizenship,) 
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Efforts are also being made to develop measures of authoritarianism (F), 
self-esteem, and need achievement. 

Attitude research with children is never easy, perhaps least of 
all when we are dealing with such sensitive areas, and when the results 
have to be comparable across countries. Comparisons between different 
age-groups will also be difficult, since attitudinal items suitable to 
one age-level may not be suitable to the next. Moreover, as we have 
seen, children develop their political percepts and awareness only 
gradually, and we must avoid the danger of asking questions concerning 
issues or attitudes which the child has not yet grasped. 

We hope that the instruments which we are developing will produce 
not merely quantitative scores to make cross-country comparisons possible 
and meaningful, but also qualitative scores or profiles which will show 
differences in the way that children see their communities, at different 
age-levels in different countries, the different emphases they place on 
certain sub-areas, and so on. In addition, we hope to relate these re- 
sults to the politicization determinants, to be able to account for some 
of these qualitative differences. 

All this will require careful analysis — much of it from the typed 
out responses rather than from computer output sheets — and, where neces 
sary, further pre-pilot runs on selected instruments. The final set of 
instruments which will emerge should be much shorter, and will have known 
statistical characteristics in several countries. 

7 . 7 Interrelation_between_Cognitiye_and_Affectiye_Domains : 

For the moment, the development of the tests and questionnaires 
in the Cognitive and in the Affective domains is proceeding separately, 
though there is a continuous exchange of information, of comments, and 
of criticism. We are not, however, losing sight of the eventual need 
to arrive at a more integrated picture of the child's political sociali- 
zation — perceptual, cognitive, and affective. Since this is likely to 
raise considerable methodological and analytic difficulties we shall, in 
due course, need to mount a small, special pilot study to see how our 
various sets of scores and results can best be brought together. 

This special pilot study cannot take place until the various in- 
struments have been much refined and shortened, so as not to overburden 
the children. Knowledge and attitudes may develop both separately and 
together, in ways that are isolated or integrated to varying degrees; 
we hope that this pilot study will show us how these processes of de- 
velopment in the child can best be assessed and expressed. 

7.8 Status of Instruments 



Many delays have been experienced with the prertesting. 

Final cognitive instruments will be ready by April, 1969 and affective 
instruments in October, 1969. The instruments given in Volume II, 
Booklet 5, are the pre-test versions. 
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CHAPTER 8 



Development of Questionnaires and Attitude and 

Descriptive Scales 



8 . 0 gyE STIOHNAIRE S_ 

The purpose of the various questionnaires is to collect informa- 
tion on background variables about the individual students being tested, 
the teachers or the schools involved in the testing program. In a 
cross-sectional survey as the present one , these variables serve as the 
independent variables or "inputs" for the analysis. 'The IEA Council 
decided that such information would be collected from individual stu- 
dents about their home backgrounds, subjects being studied at present, 
etc.; from all teachers teaching particular IEA subjects in a school 
about their training, use of teaching time, etc.; and from the school 
principals about the school organization, financing, etc. The specific 
information to be collected, of course, depends on the hypotheses for- 
mulated either explicitly or implicitly for the research. The inter- 
national subject area committees formulated many hypotheses both among 
their own members and with the aid of national subject area committee 
members. Hiese hypotheses were subject matter orientated. The IEA 
Council formed from among its own members a Hypotheses Committee, whose 
job was to review the literature both in terms of theories of education 
and in terms of previous research, so as to formulate other hypotheses, 
most of which transcended subject matters and were to do with the struc- 
ture of educational systems and various educational practices. At the 
same time IEA was responsible for the convening of a special conference 
called the "Cross-National Conference on Education, Manpower and the 
Economy " where social scientists from different disciplines were 
brought 'together to examine the results of the IEA Mathematics Study 
and to study hypotheses which could be tested in such a large scale 
survey on the theory of their own discipline. They were asked also 
to study the variables which they thought should be included in the 

IEA Study. 



The variables (or in some cases general ideas) produced by these 
three sources (subject matter committees, hypotheses committee, and the 
cross-national conference) were fed into an IEA Questionnaire Committee, 
consisting of Professors Hus^n, Bloom, Wolf and Dr. Postlethwaite. These 
variables and ideas were reviewed in terms of: 

1) translatable into paper and pencil items, 

2) suitability and acceptability for a large scale survey of this 
kind. 



After the variables had been screened, those remaining were trans- 
formed into questionnaire items. These were checked and reviewed by 
various IEA members and then assembled into draft questionnaires for 
pre-testing purposes. 

A set of notes was written to accompany each pre-test question- 
naire to indicate to National Centers the various problems which arise 
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with the translation of certain items into other languages. Where cer- 
tain technical terms had been used, these were specifically defined, 
e.g., comprehensive school, intra-school grouping, etc. 

Most of the items were left in open-ended form in the pre-test 
drafts, so that once the frequency distributions were known, the re- 
sponses to the items could be cast into a multiple choice format. Since 
the school questionnaire was particularly long, and it was thought that 
school principals might object to taking so much time to complete it, 
it was suggested that twice the number of principals required should be 
taken and the questionnaire split into two halves such that one-half of 
the principals were given the first half and the other half the second 
half. 



One concern in the present phase of the IEA Project arises from 
the proposed inclusion of ten-year-olds as a population for research. 
While the participating IEA countries have felt that the study of a pri- 
mary school population afforded great potential benefits, there was 
serious question raised as to the feasibility of testing these younger 
students. Particular concern was raised about the ability of ten-year- 
olds to provide accurate responses to questionnaire items. Consequently, 
it was decided that a pilot study be undertaken to determine whether 
ten-year-olds could provide accurate responses to such items. The de- 
sign of the study was simple. Several classes of ten-year-olds in five 
countries were administered a short background questionnaire. Follow- 
ing this, an interview was conducted with the mother of each student 
tested or a questionnaire was transmitted to the home to be completed 
by the mother. In either case, the same questions that were put to the 
student were put to his or her mother. Both parent and child responses 
were coded and punched onto Hollerith cards and sent to a central pro- 
cessing site. The countries which participated in this pilot study were: 
Australia, Sweden, Iran, Italy and Finland. In each country, an effort 
was made to sample ten-year-olds from upper, middle and lower socio- 
economic levels. Also, roughly equal numbers of boys and girls were 
tested in each country. 

The results of this pilot study were fairly clear cut. Those 
questions which asked the student to describe some aspect of his present 
life situation showed a very high degree of agreement with the responses 
given by his mother. Thus, the proportion of agreement between parent 
and child about the father's present occupation ranged from 70 # in 
Sweden to 88# in Iran. The median was 8**#. Similarly, the proportion 
of agreement on the place where the child studied in the home ranged 
from 62# in Iran to 80# in Sweden with a median of 67#* The median 
correlation between the child's age as reported by himself and by his 
mother was +.70. Thus, there was substantial agreement between mother 
and child on variables related to the present life situation of the child. 
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There was considerably less agreement between the mother and child 
on items which were retrospective or prospective in nature. For example, 
the median correlation between mother and child on the amount of additional 
schooling desired for the child was +.35 while the correlation for length 
of attendance in a nursery school was +.44 for those that had attended. 
However, the proportion of agreement between mother and child as to whether 
the child had actually attended a nursery school ranged from 83 % in Aus- 
tralia to 95# in Finland with a median of 91#* 

The fact that mother and child agree in their responses to certain 
items about the home situation is not conclusive proof of the truth of 
the response. It is, however, a reassuring indicator. On the basis of 
the results of the pilot study it was decided that it was indeed feasible 
to ask ten-year-olds to complete questionnaires that required them to 
furnish information about their present life situation both in school 
and at home. However, the number of items in the questionnaires that 
asked the student to give information about the past or the future should 
be kept to a minimum. 

The pre-test questionnaires were administered between February and 
July 1968 and the frequency distributions were sent back to IEA, together 
with a report on the pre-testing of the questionnaires and comments on 
the particular items. It would take too much paper and be of little 
interest to show the frequency distributions, even of one questionnaire 
in one country, but it is of interest to note how countries carried out 
the pre-testing of the various questionnaires. The following are selected 
excerpts from the report of the pre-testing of student questionnaires in 
Chile: 



8,1 Example of Pre-Testing in Chile 




The sample for this population included only students who are at 
present in the 3th grade of their basic education and having an age cor- 
responding to the definition of I,E,A, (coded 120 to 132), 

For the General Section, Science and Reading Comprehension Ques- 
tionnaires, students from four schools were tested. All the schools were 
urban and of Santiago, We only tested the 3th grade, because in this 
type of school, most 10 years old belong to this level. 

Socio-economic status was also considered important and we included 
students representing different socio-economic standards in this propor- 
tion: 43,3# belonged to a low socio-economic level and 38,7# were of 

middle socio-economic standard. 

As for sex, 31*2$ of the group tested were girls and 68,8$ were 

boys. 
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Included students between 14.0 and l4.ll years of age t at present 
in the 9th grade (first year of secondary education) all those students 
whose age did not fit this age definition were discarded from the final 
analysis • 

We included both public and private schools and we tried to repro- 
duce the reality of the socio-economic status of Chilean students at this 
level. Generally speaking, the socio-economic status of the students of 
this age is a little higher than that of students belonging to popula- 
tion I. 
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Sex TION 

NA1RES 


GENERAL 


SCIENCE 


READING COMPREHENSION 
AND LITERATURE 


Boys 


57.256 


63.3* 


49.8* 


Girls 


42.856 


36.7* 


50.2* 


TOTAL 


100.0# 


100.0* 


100.0* 



SPEula, tion_ IV JLS tudent^ in — their^jce^universjs.ty. y.ear}_ 



The questionnaires for this population were given to the appro- 
priate students as defined by l.E.A. 



The total number of schools tested was five schools, all of them 
public. 



The following table reflects the distribution by sex on the sampled 



Sex TION 

NAIRE 


GENERAL 


SCIENCE 


LITERATURE & 
R. COMPREHEN. 


FRENCH 


ENGLISH 


II 


68.5* 


33.1# 


34.0# 


39.3# 


28 o 256 


M 


31.5# 


66.9# 


66.0# 


60.7# 


71.8# 


TOTAL 


100.0* 


100.0# 


100.0# 


100.0# 


100.0# 
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The translation of the Questionnaires was done by three teachers 
of English. The revision of each translation was performed by a Com- 
mittee - generally speaking - formed by the same persons working at the 
qualitative analyses of the tests. The administration of the Question- 
naires for populations I, II and IV was left in hands of the teachers of 
classes tested. For population I, the General teacher undertook the ad- 
ministration of the three Questionnaires* For populations II and IV, the 
teachers of each area involved administered the questionnaires. 

All the students registered their answers in the booklets. 

Reviews of Pre-Testing and Use of Unsealed Variables 

The responses to each question from each country were collated at 
the IEA Center onto large master sheets. At the National Technical Of- 
ficer meeting held in September 1968 the questionnaires were also re- 
viewed and it was at this point that it was decided that for certain 
variables it was impossible to create one international coding scheme 
into which all practices could be categorized. It was decided, there- 
fore, to designate such variables as unsealed variables. In this case 
the National Center was asked to formulate in national terms the ques- 
tion which would obtain the necessary information for the variable. At 
the same time the National Center had to develop a classification scheme 
for the coding of the responses. The number of categories allowed was 
given by the International Center. Wherever an unsealed variable was 
to be used, the National Center had, of course, to return the classifi- 
cation scheme to the IEA Central staff. An example of an unsealed vari- 
able is father's occupation. This did not work well in the Mathematics 
Study and it is felt that the system of unsealed variables will allow 
greater flexibility and at the same time more meaningfully interpretable 
data. The question of how to deal with such variables in the analysis 
arises, since the categories used in different countries will be dif- 
ferent, but on the assumption that a single Procrustean mould would be 
likely to mask their effects in the analysis, it is agreed that it will 
be better to use the categories as they actually exist in each country. 
Thus, suppose that in country M there are four categories of teacher 
education, A, B, C and D. In the i fc ^ school in that country they will 
be represented in the proportions p/^, pg, p^, Pd* D can be taken as 
standard, and p^, pg, and p^ put in the regression betweeen schools, to 
obtain the corresponding terms in the regression equation: 






Y/sy b A p */a A + b B pb/sb + be PC/«C 

in standard form or alternatively 

Y (b A sy/s A ) p A + (bs 8y/s B ) PB + <b C Sy/s<j) p c 



in metric form. 



Furthermore, provided A, B and C are kept together as a group in the re- 
gression, and are not allowed to be separated by other variables through 
the use of descending partial®, the addition to "Sr made by the group can 
be obtained when it is in any position, including the first and the last, 
with respect to other groups of predictors. 



Consequently comparing the effects of teacher training in differ- 



ent countries can give 








Country M 


N 


0 


etc. 


b A 


b E 


b J 








b K 


etc , 


b C 


b G 


«!• 




- 




- 




This is in standard form. 


Alternatively the 


same table could be given 



in metric form. 



It is also possible to give 
Country M N 



Here k 2 is the addition made to B 2 by the group of teacher training cate* 
gories when the group is taken first, last, or in any other position in 
the regression thought fit. 

There is no need to have the same number of categories in all coun- 
tries, The number of regression coefficients will always be one less 
than the number of categories, but there will always be a single kr, 
which will be maximised by taking the categories as they actually exist 
in each country, instead of trying to force them into a single mould. 



0 



etc, 

etc. 
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The results of the pre-testing were reviewed and the items for 
the final questionnaire decided upon. All of the responses to items 
were then closed on the basis of the frequency distributions obtained 
in the pre-testing, except for questions such as age, grade, father s 
occupation, and expected occupation, which were to be given by students 



It was felt that ten-year-old students could not be expected to 
fill ir their replies on an answer card and they will fill in their an- 
swers on the answer booklet and these will be punched onto punched cards 
afterwards by the National Centers. The same applies to the school 
questionnaire, but the reason in this case is that some of the 
would require four columns of answers and therefore it was much simp 
for school principals or their deputies to fill in their answers ^ rect ^ 
on the booklet. In all other cases the answers will be put directly onto 

the answer cards to be used by IEA, 



The examples of the final questionnaires and coding and punching 
schemes where appropriate, together with accompanying notes, are in- 
cluded in Volume II, Booklet 4, 
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8.3 Attitude and Descriptive Scales 



I 



The construction of attitude scales with international validity presents 
some very serious methodological problems. Firstly, there is the problem 
of defining the dimension that the scale is intended to measure in terms 
which are equally meaningful in the different cultures for which use of 
the scale is envisaged. 



There is a problem inherent in the translation of attitude state- 
ments from one language to another. Although it is usually possible to 
make a fairly exact translation of the denotative meaning of a statement, 
it is often difficult to do this without considerably increasing the 
nature and complexity of the statement from a linguistic point of view. 

It is even more difficult to also translate the connotative implications, 
so important in attitudinal work, from one language to another. 



Cnee an adequate scale has been drafted, it is necessary to validate 
it within each of several different countries, Unfortunately the criteria 
for validation (even content validation) may appear very different to 
the recognized local experts in each culture. These views must be 
reconciled and a compromise achieved. 



Finally, there is the problem of "acceptability." Many Western 
European countries and others are much less used to attitude scales 
than is the United States and the use of scales which seem quite 
innocuous in the United States may be vetoed in other countries by 
the researchers themselves, the Ministries of Education, the teachers' 
unions or even the school principals. 



Despite these very great difficulties, it was decided that I.E.A. 
should persevere with the development of suitable international scales. 
The "replications-over-countries" structure of the I.E.A. project 
meant that even if a particular scale was administered in only a few 
countries, the results of this could still be valuable. I.E.A. analyses 
are by and large carried out on a country-by-country bais. 



It has been suggested that although we seek to measure the same 
dimension in different: countries, it would be possible to do this by 
using different sets of items. It seems clear that some items are 
more appropriate in some countries than in others. A possible procedure is 



a) 

b) 



c) 



d) 

e) 



Carry out factor analyses within each country 
Match factors across countries to identify the one 
that is of interest 

Choose for each country the set of items that seems 
most adequately to measure this factor 
Establish norms for these different scales 
Administer the different scales in different countries 
and standardize the results 
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This work on the scale construction has been carried out during 
the period from September, 1967, until the present. Unfortunately, 
the need to produce scales by the end of Stage 1 (December, 1968) 
together with the considerable variation of school schedules (and 
therefore of possible testing occasions) in the different countries, 
made it impossible to carry out the full series of steps outlined 
above in all cases. The basic shortage of time wias aggravated in 
some countries by school strikes, committees of revolutionary 
students, unexplained postal losses, and the like. We have had, 
therefore, to take certain shortcuts particularly with regard to 
Step 2, but it is emphasized that no scale will be considered to be 
in its final form until it has been pre-tested in all the relevant 

countries . 

The statistical analyses of the pre-test data for these scales 
have all been performed by the I.E.A. data processing unit which 
has been stationed in London during this period. The analysis that 
has been performed is of two different types: a) a factor analysis to 
investigate the dimensionality of the attitudinal space concerned, 
and b) a standard item analysis giving response distributions for each 
variable together with internal consistency figures • 



,4 General Attitude Scales 

These scales are intended to be administered to all students 
involved in the I.E.A. study: 

1. T.ik e/Dislike School Scale . After an initial trial of an 
open-ended instrument in England, a pool of some thirty statements was 
formed. These statements were put into two forms: multiple choice or 
the Likert type and true/false. Further piloting in England suggested 
that the true/false items were working better and ten of these were 
chosen for full pre-testing. Results so far analyzed from seven 
countries indicate that the scale is reasonably satisfactory. Re- 
liability estimates had a median value of .80. A slight rewording was 
proposed for one of the items and two other items proposed by National 
Centers have since been added so that the present form of the scale 
contains twelve items. In this form a scale will be administered during 

the dry-run. 

2. F-Scale. The desirability of including such a scale in the 
battery was discussed during the spring of 1967. Some piloting work 
was carried out during 1968 in England using items taken from the 
work of Adorno, Franl-Brunswik and others. A draft scale for inter- 
national use was sent to several National Centers for administration 
and to all National Centers for comment. There was a great deal of 
criticism of this particular scale and indeed of the suggestion that 
such items be used at all in research projects like I.E.A. As a 
result of this, the matter was discussed by the I.E.A. Council at 
its meeting in Hamburg during December, 1968. At this meeting it 
was resolved, because of the good-will that I.E.A. might lose b Y 
administering this type of scale in many of the participating countries, 
that further work on the F-scale be stopped. Thus, no F-scale will 
appear in the final I.E.A. battery. 
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The problem with this arises in step b) . Factor matching is 
difficult to accomplish, and in the circumstances of I.E.A. there would 
be very little evidence on which the matching could be based. However, 
a firm base for matching would be important because I.E.A. would clearly 
be open to attack when the results were reported, and it was clear that 
different items were used in different countries. It seems that we would 
not be justified in adopting a policy of varying the items between 
countries unless we had very sound substantive or statistical evidence 
proving the equivalence of the scales so formed. 



For these reasons the I.E.A. decided to concentrate on the develop- 
ment of scales which were directly translatable into the different 
languages and cultures involved in the project. However, a degree of 
flexibility was left to cope with local translation difficulties. Although 
a single international form for each item (together with translation 
units) was produced, individual National Centers were allowed to vary 
the wording of particular items if local circumstances so required. It 
was not expected that any National Center would modify more than about 
5% of the items in this way, and these slight local variations were 
thought unlikely to affect the general international validity of the 
scales. 



A set of procedures was developed for the construction of such 
scales. This comprised three separate steps: 



Step 1. 



The development of a multiple-choice item pool from 
data gathered directly from students. Frequently, 
this material would come out of interviews or from a 
questionnaire of the sentence completion type. In most 
cases this work was performed within one country. When 
a suitably large pool of items was developed, they were 
given a preliminary trial with students of the appropriate 
age but still within the one country. 



Step 2. 



From these results a draft scale of perhaps twenty items 
was produced for pilot testing in two or three I.E.A. 
countries with different languages and cultures than the 
country of origin. 



Step 3 . 



The evaluation of the statistical performance of the 
items together with the detailed comments gathered 
from the National Centers who had conducted the pilot 
testing led to a revised scale which only then was put 
forward for pre-testing in all I.E.A. countries. It 
was expected that this pre-testing would uncover .minor 
problems of translation in cartain countries but would 
not lead to more than local revision of a few items. 



8-10 






m 



a •mh*x nh “ 



igaaai«iiaaaaiffigiffiiaafiaiaaafl£«ias^ i wM 



4. Self-esteem Scale. Pilot work was carried out in England 
with both a true /false and a multiple choice form, each developed from 
a pool of attitude statements about the self. Of these, the true/false 
scale seemed to be superior but the results were in general very 
unsatisfactory. After discussion by the I.E.A. Council in Hamburg, it 
was decided that as this scale was not regarded as crucial for the work 
of the Association, it should be dropped from the test battery. 



8.5 Science Attitude Scales 

The International Science Committee proposed two types of attitude 
scale: one, a behavioral attempt to approach "Interest in Science." 

Pilot work carried out in England in collaboration with an "Attitudes 
to Science"project of the National Foundation for Educational Research 
in England and Wales suggested that two quite separate dimensions of 
attitude could be discerned within the latter . In simple terms there 
were an abstract view of science as one of the important agents for 
change in the world, and the more personal view of science as a subject 
for study in school. Although the testing of these dimensions was 
i originally tried out with students from Populations I, II and IV, it 

seemed clear that they were not working satisfactorily with Population I 
(who, for the most part, had not had any formal study of science in 
school). For this reason, the final forms of these two attitude scales 
are intended for use with students aged 14 and upwards. 
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1. Science in the World. In this scale science is seen as a 
force for good or evil in rather abstract terms. The items that 
formed the draft scale for pre-testing were chosen following an 
English factor analysis on a pool of some 300 attitude statements. A 
six item scale was tried out in twelve of the I.E.A. countries and 
the results seemed very satisfactory. The twenty-four reliability 
estimates for Populations II and IV had a median value of 0.68 whiqh 
seems to be adequate for a survey such as this. The median of the\ 
eight estimates tested are proposed for use in Stage 2 without further 
modification. 

2. Science and the Self . This scale deals with the personal 
appreciation of science as a subject for study in the school. The 
development of this scale followed the procedures outlined for 
"Science in the World." The reliability estimates which emerged 
from the pre-testing had a median value of 0.71. Once again, the 
scale is being retained in its pre-testing form for its administra- 
tion in Stage 2. 
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3. Interest in Science . This scale is intended for use with all 
populations. The International Science Committee produced a lengthy 
list of scientific and science-related activities which seemed appropriate 
for children. To this were added questions concerning school performance 
in science subjects as compared with other subjects, and questions 
designed to investigate the extent of the reading of scientific materials 
outside the classroom. After pilot testing in Finland, Japan, and the 
United States, the original pool of some sixty items was reduced to 
one of twenty-four. The revised scale was then circulated to all 
National Centers for pre-testing. The results for the ten countries 
which have so far been analyzed suggest that apart from a few minor 
changes in wording in certain countries the scale is quite satisfactory. 
The reliability estimates with Population I are about 0.7 and with 
Populations II and IV about 0.8 and above. 

8.6 Literature Attitude Scales 

The scales were proposed by the International Committee on 
Literature. Their development is discussed below. It must be re- 
membered that the Literature Committee did not concern itself with 
tests or scales for Population I. 



1. Transfer_of_Affect. This scale seeks to measure the extent 

to which” tudents”see”£hemselves as participating in the fictional liter- 
ary situations they encounter, and the extent to which fictional charac- 
ters are imagined to play a real part in the students' lives. We are 
concerned, therefore, with the amount of transference which takes place 
between the student and the literary word he confronts. We are inter- 
ested in the extent to which he interprets reality in terms of his 
literary experiences. Twenty-one items were prepared by members of the 
committee working in the United States. These were given a field trial 
in Belgium and Iran and, as a result, a revised scale of ten items was 
tried out in four countries: England, Sweden, Iran and Finland. The 

median reliability for a scale comprising these ten items was estimated 
at 0.63* The Literature Committee, after discussing the results, pro- 
posed minor alterations to one of the items and after making this 
change, the items were presented for use in the dry-run. 

2. _ — — — — 2— — — — C * This was a scale proposed by the committee to 
investigate”he”tudent* s attitudes toward censorship in its various 
forms. Twenty-nine items were drafted by committee members in the United 
States and were circulated to National Centers for pilot testing. It 
unacceptable and, elsewhere, serious doubts were expressed as to 
whether such an instrument could be used in the schools. In view 

of these criticisms, the committee reconsidered the matter and 
decided to withdraw the scale. No further work has been done. 
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3, Interest in Literature* The scale seeks to measure the ex- 
tent to which“iSe"siu3ents’rea3’5ooks for pleasure and engage in other 
literature — related activities. The twenty items originally drafted in 
the United States were piloted in Belgium and Iran. As a result of this 
trial the scale was reduced to thirteen items with substantial rewording 
in some places. Pilot testing was then carried out in England, Sweden, 
Iran and Finland with moderately satisfactory results. The median re- 
liability was 0.37* Various suggestions were made to improve the struc- 
ture and wording of some questions in an effort to improve the reliabil- 
ity. These changes have been made in the form of the scale proposed for 
the dry-run. 
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The International Committee on the Study of French as a Foreign 
Language requested that work commence on three attitude scales. They 
were designed to help answer questions about the way in which a student’s 
personality structure could influence the learning of French, and, con- 
versely, how his study of French could affect his general attitudes 
towards French people and culture. A first scale, Ethnocentricism, was 

composed of items taken from the work of Adorno and others. The word- 
ing had been changed in some places to make the items more appropriate 
for a cross-cultural study. A second scale was "Attitudes Towards the 
French People" and these items came in part from the work of W.E. Lambert 
and others in Canada. The third scale, "Preference for Own Country Over 
France" owed much to Canadian work and was supplemented by items pro- 
posed by the International Committee. A draft instrument of 29 atti- 
tudinal statements was formed and this was tried out in Sweden and Scot- 
land. The results were generally rather poor with the reliabilities for 
subscales being of the order of 0.5* This was disappointing since use 
of many of these items in Canada and the United States suggested that 
reliabilities of 0.80 and above should be attainable. 



A new revised instrument was composed in which the attitude 
"Preference for Own Country Over France" was excluded. This was done 
because of the apparently high correlation between this scale and the 
"Attitudes Towards the French People." The remaining items were con- 
siderably revised and the resulting instrument (6 "Ethnocentricism" 
items and 12 "Attitudes Towards the French People" items) was circu- 
lated for pre-testing in all National Centers. The results from four 
countries have now been analyzed and the findings are generally satis- 
factory (median reliability for Ethnocentricism; median reliability for 
"Attitude Towards the French People." 



Concurrently with this work, it was suggested that the Semantic 
Differential technique be used to investigate more directly the national 
stereotypes involved. Accordingly, National Centers were asked to pre- 
test an instrument in which "Self," a "Typical French Person," and a 
"Typical Person of ray Own Country" were analyzed. These three were rated 
on a set of 12 adjective pairs. Analysis of this data and the continuing 
development of other scales is now proceeding. The decision concerning 
whether all scales will be used in the French testing has not yet been made, 
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8.8 English as a Foreign Language Attitude Scales 



It was at first thought possible to construct scales which were 
in general similar to those requested by the French Committee • A com- 
plication, however, was that English-speaking people comprise several 
different cultural groups and it was considered probable that in some 
countries attitudes towards British people would be very different to 
attitudes toward Americans. A draft instrument for testing "Ethno- 
centricism" and the "Attitude Towards English-Speaking People" was pilot 
tested in Italy and in West Germany. The Italians produced two forms 
which separately measured attitudes towards British people and attitudes 
towards American people while the Germans kept the draft instrument in 
its original undifferentiated form. Results from these pilot testings 
were very disappointing and strong criticism was received from the 

Italian National Center since what they regarded as pseudo-political 
questions were being asked. The English Committee and the I.E 0 A. Bureau 
when discussing this matter concluded that the attitude scales should 
not become involved in political issues and, in consequence, decided to 
drop scales which contained references to English-speaking people. After 
consultation between the data processing personnel in London and the 
chairman of the English as a Foreign Language Committee, a new pool of 
47 attitudinal statements about English was constructed. These deal 
specifically with the study of English in schools and with the English 
language as a cultural entity. This draft instrument was sent to all 
participating National Centers and the data which is now being returned 
will be subjected to a factor analysis before a final form of scales can 

be proposed. This scale will be a National Option. Scales of interest 
in English as a foreign language and of attitude towards learning 
English as a foreign language in school are also being prepared. 

8.9 Civics Education Attitudes 

These will not be dealt with here and readers are referred to 

Chapter 7 

8.10 Descriptive Scales 

The purpose of the descriptive scales is quite separate from that 
of the attitudinal scales described above. These are intended as measures 
of the school environment. It has been found (by Pace and Stern as well 

as by previous I.E.A. studies) that there are some aspects of the school 

environment for which one can get better data from the students them- 
selves than one could hope to get from the principal. We can ask the 
principal to report on the enrollment of his school and on the academic 
qualifications of his teaching staff but we cannot get a reliable assess- 
ment from him as to the authoritarian or permissive atmosphere that is 
characteristic of his school. However, if we collect data from a group 

of students in school and then average it, we come up with a measure of 

at least moderate reliability and validity for the school as a whole. 

The instruments themselves are similar in appearance to conventional at- 
titude scales and are presented to the students as such. 
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Hie criteria for success of these scales are that they should 
discriminate between schools and be consistent within schools. This can 
be expressed most simply as a F-ratio of between to within school variance. 
In the development phase of the scales, F-statistics are worked out for 
each item separately within each population and each country, and in 
general, items which have consistently high F values (say 3 or above) 
are retained. Similarly, F-ratios calculated for the total score on a 
scale gives a measure of the general effectiveness of the scale and for 
the I.E.A. work, a target of F is = 10 was set for each scale. 

8.11 General Descriptive Scales 



Two environmental measures were proposed. One, an "Authoritarian- 
Permissive" scale and the other on a "Subject Oriented-Student Oriented" 

( dimension. After pilot testing in England, an instrument of items cover- 

[ ing these two scales was pre-tested by National Centers and the results 

; were subjected to Analysis of Variance, Factor Analysis and a standard 

Item Analysis. It became clear that the dimension "Subject Oriented- 
Student Oriented" was meaningful only in England and Scotland in the 
| sense that these were the only countries in which there appeared to be 

I real distinctions between schools. The scale dealt with characteristics 

| such as: (a) the students change classrooms to meet a particular teacher 

I rather than vice versa, and (b) the availability in the classroom of some 

I non-standard subjects for students with particular interests and needs. 

I These needs appear to be more a consequence of national education policy 

I than of the particular school regime. The chief problem with the au- 

| thoritarian measure was that practices such as "students standing up 

f when a teacher entered a room," which proved highly successful in dis- 

I criminating between schools in some countries, were regarded as socially 

| dangerous in other countries where the practice of standing for a tea- 

cher was universal. "The pupils 1 standing up has nothing to do with 
I authoritarianism because any other behavior would be extremely bad- 

| mannered." It was apparent, therefore, that the items for this scale 

I must be chosen with great care and that there might have to be local 

variations in order to have effective items in every country. A revised 
| scale of 12 items has been proposed for use in the dry-run and, using 

| the pre-test data as a guide, it is estimated that this scale should | 

} achieve F-values of between 8 and 10 on average. 

i' 1 
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Two descriptive scales were proposed by the International Science 
Committee. The first covered the classroom teaching with the textbook 
or lecture at one end of the continuum and an experimental approach using 
demonstrations, laboratory work, field work, etc., at the other. This is 
a fairly straightforward scale of 12 items which achieved a median F-value 
of 8.3 during pre-testing. 

The s'econd scale which deals with the nature of laboratory work 
is applicable only in those schools which have science laboratories. At 
one end of the continuum is a laboratory session in which the students 
are given detailed written instructions which cover all the work that 
they are allowed to do and at the other end is a laboratory in which the 
students design and carry out their own experiments without intervention 
from the teacher. This scale has 8 items. It appeared to be satisfac- 
tory in pre-testing and has come forward without modification for use in 
the dry-run. The attitude and descriptive scales are presented in 
Volume II, Booklets 4 and 5. 
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CHAPTER 9 



Hypotheses. Sampling Designs and Between Country Analyses . 

9.0 Development of Hypotheses 

The IEA Project can be conceived of within the general frame- 
work of an empirical approach in comparative education. Until 
recently comparative education confined itself to qualitative 
descriptions of various educational systems and to analyses which 
then were limited by the available data. Comparative education 
should, of course, try to achieve what has long since been accomplished 
in other social sciences, namely to adopt the log ico -deductive way 
of reasoning and the empirical testing of the hypotheses advanced. 

In comparative education a set of systems of education, 
as represented by a group of countries, can be regarded as a large 
•'laboratory" where a wide range of structural and instructional 
practices are employed. By relating the outcomes of the systems, 
in terms of specific knowledge, skills, attitudes, etc., to 
different groups of input factors, such as economic standards 
and cultural level, teacher competence, equality of opportunity, 
etc. , countries can learn from each other. There are at least 
three basic prerequisites for such studies. Evaluation instru- 
ments applicable at the international level will have to be 
developed. Representative samples of students at certain age or 
grade levels will have to be drawn in all the participating 
countries. The instruments will have to be administered and 
scores uniformly in all the countries. 

In the present phase of the IEA Project hypotheses have 
been generated in two different ways. The international subject 
matter co mm ittees have been asked to list hypotheses which are 
closely related to their particular subject fields. Thus, for 
instance, the reading comprehension committee listed a set of 
predictor variables which were hypothesized to account for 
differences in reading performance both within and between 
educational systems. The list of suggested questions and 
hypotheses by the subject matter committees are indicated in 
Appendix I. 

The most important work, however, in trying to advance 
hypotheses thought to be independent of subject fields was 
carried out by a Hypothesis Committee. which attempted to draw 
its hypotheses from previous work in the field, from its previous 
conceptual framework (Husen, 1967) and from the suggestions of consultants 
researchers in the different National Centers. 

On the basis of the experiences gained during the mathematics 
phase of the IEA Project, it was suggested that a more powerful 
conceptual framework for the study should be developed. The 
mathematics phase was designed entirely by educational researchers 
which tended to overemphasize school and teacher variables in the 
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list of predictor variables which were drawn up and turned into instru- 
ments used to test the hypothesis. The educational system must be regard- 
ed as a sub-system of the social system at large. In order to develop 
an input-output model, which hopefully could help in the development of 
fruitful hypotheses, sociologists, economists and political scientists 
convened with representatives from the IEA group on two occasions in Ham- 
burg in 1966, and at Lake Mohonk, N.Y. in 1967 respectively in order to 
suggest a better theoretical model for the IEA project and to list varia- 
bles within the social matrix, which were thought to be relevant to the 
outcomes of the educational system. Both cognitive and non-cognitive out- 
comes of these systems could be related to various groups of input varia- 
bles, such as expenditure per pupil, teacher characteristics, equipment, 
etc. (Cross-National Conference on Education, Manpower, and the Economy, 
1967). 



Previous research carried out, for instance, by the Plowden Com- 
mittee in England and the Coleman group in the United States, seems to 
indicate that the major portion of inter-individual differences in school 
attainments are accounted for by non-scholastic factors, such as home 
background and parental motivation. Since the socio-cultural variability 
in the present phase of the IEA Project is much greater, hopefully a 
general answer will be obtained to the question of how close the rela- 
tionship is between the socio-cultural matrix and the "productivity" of 
the educational system. An answer to this question has important impli- 
cations for educational policy, particularly in the developing countries, 
because if education cannot achieve considerably more than the ceiling 
set by the socio-cultural level of a particular country, the place of 
education in the list of priorities would have to be revised. 

It should be pointed out that the second phase of the Project 
offers a unique opportunity to test certain general hypotheses on the rela 
tionship between certain input factors and the outcome of the system, not 
only across countries representing ^a wide variety of socio-cultural pat- 
terns, but also across subject areas. 

In addition to the specific subject matter hypotheses and ques- 
tions, (See Appendix I), the hypotheses committee formulated hypotheses 
which transcended a particular subject. In doing this, it stated the 
hypotheses, indicated which subjects were relevant, the populations for 
which it should be tested (1,2, 3, 4,), and whether the analysis should be 
between students within schools (ST) , between schools (SC) , or between 
national systems (N) . 

These hypotheses will need additional work before the details 
of data processing can be executed. Insofar as possible, composite 
hypotheses will be formed from the single hypothesis for the purposes 
of hypothesis testing. Further committee effort will be needed to com- 
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plete this phase of our work. The specific hypotheses are listed below 
and are grouped to indicate whether they refer primarily to the Home 
and Community background, to the Educational Policy of the System, to 
the School, to the Curriculum and Instruction, the Teacher, or primarily 
to specific characteristics of the students. 

Home and Community Hypotheses 

1» Achievement varies with parental socio-economic status, 
least in the case of the simple skills and content , most in 
the case of the complex . 

(All subjects) (1,2, 3, 4) (ST, SC) 

2. There will be systematic socio-economic differences bet- 
ween students in 14-year old populations and those in pre- 
University populations. 

(Not applicable) (2 and 4) (N) 

3. Achiev emen t of urban pupils will exceed that of rural 
pupils. 

(All subjects) (1,2, 3, 4) (ST, SC) 

4. Achievement, interestes attitudes will be positively 
correlated with social status of pupils' houses. 

(All subjects) (1,2, 3, 4) (ST, SC, N) 

5. Motivation is positively related to the discrepancy bet- 
ween father's occupation and expected (aspired) occupation. 
(Science, English) (3 and 4) (ST, N) 

6. Pupils whose native language is different from the language 
of instruction will have lower achievement scores in all subjects 
than pupils for whom there is no difference. 

(All subjects) (1,2, 3, 4) (ST) 

7» Level of achievement in foreign languages is related to fami- 
ly community and school attitudes towards the language studied. 
(French, English) (1,2, 3, 4) (ST, SC, N) 

Level of achievement in a foreign language is related to op- 
portunities for contact with persons and groups using that lan- 
guage. 

(French, English) (1,2, 3, 4) (ST, SC, N) 

Educational Policy of the System Hypotheses 

1* The level of achievement at age 14-14.11 is not related to 
the mean age when compulsory schooling begins. 

(Science, English, French, Reading Comprehension) (2) (N) 
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2. When equal proportions of the age group are compared, 
countries will not differ in their terminal level of achieve- 

ment . 



(All subjects) (4) (N) 



3. Average level of achievement in the terminal group will be 
lower in countries with larger percentage of age group still 
in school. 

(All subjects) (4) (N) 



4. Mean performance on total test in any given subject area of 
top 5 per cent (10 per cent) of the age group is higher itt sys- 
tems with high retentivity than in systems with low retentivity 
(Science, English, or French) (3 and 4) (SC, N) 



5. Sex differences in mean performance in each subject as well_ 
as attitudes toward the subject is related to amount of co- 
education in a given country . 

(All subjects) (2,3,4) (N) 



6. The "total yield" (defined as an answer to the question 
'How many are brought how far?') is higher in a highly reten- 
tive than in a less retentive system. 

(Science, English or French) (3 and 4) (N) 



School Hypotheses 



1. Achievement » interests and attitudes in a school will be 
related to total enrolment of the school. 

(All subjects) (1,2, 3, 4) (SC) 



2. The level of achievement in a given subject is not related 

to size of class. 

(All subjects) (1,2, 3, 4) (ST) 



3. The level of achievement will be higher and the variability 
will be lower in specialized schools than in comprehensive 
schools . 

(All subjects) (1,2) (SC) 



4. School schievement will be positively connected to (a) 
the proportion of specialist teachers of the subject at school, 
(b) the provision of laboratories, (c) the provision of books. 
(All for (a), science for (b) , all for (c)) (1,2, 3, 4) (SC) 



5. Student achievement will be related to per student expen- 
diture for teachers' salaries and for other expenses. 

(All subjects) (1,2,4) (SC, N) 



6. Achievement , attitudes and interests in school subjects are 
unrelated to available funds, space or teacher qualifications. 
(All subjects) (1,2, 3, 4) (SC, N) 
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Curriculum and Instructions Hypotheses 



Level of achievement will be higher when the number of 
subjects studied is smaller. 

(All subjects) (4) (ST,SC,N) 

2. Holding opportunity to learn constant, inquiry-centered 
methods will produce higher and less variable achievement score s- 
and affective indices than will more traditional ( drill ) 
methods . 

(All subjects) (1,2, 3, 4) (ST, N) 

3 t The profile of test performance in each country will be re- 
lated to the national emphasis on each topic (as reported in 
teachers' ratings) in each school program. 

(All subjects) (1,2, 3, 4) (N) 

4. S tudent achievement in a given subject is (a) not correlated 
to number of school hours of schooling per week but (b) related 
to those of a given subject as well as (c) to number of home- 
work spent in the subject. 

(All subjects) (1,2, 3, 4) (ST) 

5. Student achievemeftt in partscore (t opics and objectives^ 
will be related differently to the number of hours per week 
devoted to all school homework as well as those devoted to the 
subject according to the kind of objectives and/or subject- 
areas . 

(All subjects) (1,2, 3, 4) (ST) 

6. Achievement is not related to teaching method or learning 
method . 

(All subjects) (1,2, 3, 4) (ST,SC,N) 

7. Methods of learning involving first hand experience 
(laboratory, primary sources) are related to interest in sub j ec t 

matter . 

(Science, Literature) (2,4) (ST) 



8. Student achievement will be higher and less variable when there 
are provisions for individual differences in learning rates and 
styles. 

(All subjects) (1,2, 3, 4) (ST, SC) 

9. Student achievement will be higher and less variable when 
the maximum per cent of teachers’ time in the classroom is spent 

in teaching. 

(All subjects) (1,2, 3, 4) (ST, SC, N) 
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10. There is an optimum amount of instruction in a second lan- 
guage - beyond that the students show little additional progress 
in achievement, interests and attitudes . 

(French, English) (2, 4) (ST) 



11. Variations i n __ the a chievement profile are related to variations 
in teachings objectives and methods. 

(French, English) (1,2, 3, 4) (ST, SC, N) 



1 ?' * n an ^ one sub J ect th e achievement, interests and att itudes 
will be related both to the stated aim’s of instruction and to 
the opportunities given for learning. 

(All subjects) (1,2, 3, 4) (SC, N) 



Teacher Characteristics Hypothes es 



1 - .fS Hgent achievement will be positively correlated to the 
quality and quantity and recency of training of teachers. 
(All subjects) (1,2,3 and 4) (ST, SC) 



2. Achievement scores will be higher in schools where teachers 
fee^. themselves and have greater freedom in determining what 
will be taught and how it will be taught. 

(All subjects) (1,2, 3, 4) (SC, N) 



3. Students whose teachers have had recent (within 5 years) 
n-service training in a subject will have more positive atti- 
re 5 to ward and interest in that subje ct . 

(All subjects) (1,2, 3, 4) (ST, SC, N) 

4. Achievement, will be related to the amount and quality of 

teacher education. y 

(All subjects) (1,2, 3, 4) (ST, SC, N) 

5 * Atti tude of teachers towards their profession and subject 

Ha tter will correlate positively to the achievement and motiva- 
tion of the pupils. 

(All subjects) (1,2, 3, 4) (ST, SC, N) 

^ * Teachers attitudes towards (or expectations about) the 

children s learning are strongly related to achievement in a 
given subject . 

(All subjects) (1,2, and 3) (ST, SC, N) 

Student Characteristics Hypotheses 



1. Motivation is positively correlated with achievement in 
any given subject area. 

(All subjects) (1 & 2) (ST, SC, N) 
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2. Total score will be positively correlated with interests 
and attitudes about the subject. 

(All subjects) (1, 2, 3, 4) (ST, SC, N) 

3. Students planning to enter vocations in which they consider 

a subject relevant will make higher achievement scores than other 
students . 

(All subjects except Reading Comprehension) (2, 4) (ST, SC, N) 

4. Pupils* profiles of scores are congruent with pupils* cog- 
nitive style, 

(All subjects) (1,2, 3, 4) (ST, SC, N) 

5. Achievement (also interest/attitudes) is correlated positive- 
ly with perception of utility of competence in teh subject for 
upward (social) mobility. 

(All subjects) (2,3,4) (ST, SC, N) 

6. Achievement, interests and attitudes are positively corre- 
lated with estimated or reported importance of the subjects. 

(All subjects) (2,4) (ST, SC, N) 

7. Students will achieve at higher levels and have more fa- 
vourable interests and attitudes when they have high esteem for 
themselves as students than when they have high esteem for them- 
selves as human beings. 

(All subjects) (1, 2, 3, 4) (ST) 

8. Students will achieve at higher levels and have more fa- 
vourable interests and attitudes when they have relevant non- 
scholastic competences. 

(All subjects) (1, 2, 3, 4) (ST) 

9. Difference between student's occupation expected (desired) 
and his parents * occupation will be related to the student * s 
cognitive outcome in a given subject . 

(All subjects) (1, 2, 3, 4) (ST, N) 

10. The difference between student's expected (desired) occupa- 
tion and his parents' occupation will be related to the student's 
interests in and attitudes towards a given subject . 

(All subjects) (1, 2, 3, 4) (ST) 



11. Level of achievem ent in a foreign language is related to the 
"linguistic" distance of the mother tongue from that language. 
(French, English) (1, 2, 3, 4) (ST, SC, N) 
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12. Student achievement, interests and attitudes will be differ- 
ently correlated with the difference between their expected 
(desired) occupation and their parents ’ occupation according to 
the type of school they attend. 

(All subjects) (1, 2, 3, 4) (SC) 

13. Student perception of his achievemen t in the given subject 
will be related to the status and kind of his occupation expect- 
ed (desired) . 

(All subjects) (1, 2, 3, 4) (ST, SC) 

14. Achievement in any one subject is positively related to par- 
ticipation in non-school (’'cultural") activities of the same type. 
(All subjects except Reading Comprehension) (1,2, 3, 4,) (ST, SC, N) 

15. The profile of achievement is positively related to the 
profile of participation in non-school ("cultural") activities 
of the same type. 

(All subjects except Reading Comprehension) (1,, 2, 3, 4) (ST, 

SC, N) 

1 6. Achievement (in any subject) is positively related to 
student evaluation of intellectual as compared with other pos- 
sible student roles. 

(All subjects) (2, 4) (ST, SC, N) 
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9.1 Sam pling Designs 
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Much progress has already been made. With three exceptions, set 
out below, all countries have adopted the general design suggested at 
the outset in Bulletin No. 3. In this design the sampling proceeds in 
two stages, the first stage being the selection of schools from a strati- 
fied frame , and the second the selection of students from selected schools. 
For schools the factors of stratification are size, type, sex and regi • 
Bach student in the population is given the same chance of e “* e ”;“ s * 

sample, by keeping the overall sampling fraction constant. Atthe same 
time the design effect is kept small by avoiding wide variation in the 
numbe^of students selected from each school. This is achieved by using 
the school measure of size. The probability of selecting the school is 
made directly proportional to this measure, and the conditional probabil- 
ity of then selecting a student from the school is made inversely pro- 
portional. In other words schools are given one, two, three or n°re 
tickets in the draw according to their sizes. When a school with o 
ticket is drawn all its students in the population are includedinthe 
sample. A two ticket school has half its students included, a three 
ticket school a third, and so on. In choosing measures of size the 
miding principle is to keep the average number of students selected per 
school^ in the neighbourhood of thirty, and to avoid wide departures from 
this average. On the experience of the mathematics study the P ri ° r ® s 
timate of the design effect from this plan is in the neighbourhood of 
four or less, so that a sample of three thousand students from a hundred 
°ZoZ should be equivalent to a single stage sample of 750 students or 
more, with standard errors in the neighbourhood of 3% of the correspond- 
ing standard deviations. This prior estimate will of course be replaced, 
during the data processing, by a posterior estimate of the design e 
obtained by the comparison of half sample and whole sample estimates, 
the half samples being drawn on the computer according to the full com- 
plexity of the sampling design. 



The strong point of this design is that the overall sampling 
fraction is kept constant even when the measures of size are very rough. 
••Size" means the number of students in the school in the relevant popu- 
lation, and when this is known accurately for all schools Size an 
"Measure of size" can be identical. But where there are no central 
statistics giving this information an approximation, such as the total 
number of students divided by the age range of the school, can be used 
instead, as the measure of size. Roughness in the measure of size will 
not bias the sample; its effect will be to produce a rather greater 
variation in the number of students selected from each selected school 
than would be the case if the measure of size were more accurate, rhis 
in turn will produce a rather greater design effect. But even a ^ ery 
rough measure of size will produce a much smaller design effect than no 

measure. 
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When the sample of schools has been drawn it will be tested by 
using it to estimate marker variables, these being variables for which 
a complete count already exists. An example from Scotland is enclosed 
to illustrate this process. It will be seen that the sample estimates 
all accord closely with the population figures, except in the case of 
very small strata, where the relative sampling variation is bound to be 
large, and also unimportant. 

The samples of schools are being drawn now, well in advance of 
the date of the field work of the survey, in order to give ample time 
for checking the correspondence between the design and its execution. 

If a relatively large number of new schools come into existence between 
the date of drawing the samples of schools and the date of the field 
work a supplementary draw will be made to cover this new population. 

Since Population 1 occurs in primary schools, and the other 
populations in secondary schools, separate samples of primary and second- 
ary schools will in any case be needed. Within the secondary field some 
countries are drawing separate samples of schools to cover each popula- 
tion; others are sampling every population that occurs within a selected 
school. The latter is the more appropriate procedure in a small country, 
where the total number of schools is correspondingly small. The main 
complication that it involves is that a school's measure of size may 
have to be a compromise between the ideal measures for two different 
populations. This is not serious, because, as was said above, the 
measure of size need only be rough, and any reasonable compromise be- 
tween the alternatives will be good enough. In extreme cases it may 
be necessary to draw a supplementary sample of schools to cover a small 
Population *V. 

Before the field work begins selected schools will furnish the 
National Center with an account of the numbers of their students in the 
relevant populations. In some countries this account will include the 
names of all such students, and the National Center will nominate the 
students to be tested, in accordance with the sampling fraction for each 
school. In other countries, for example in Scotland, this procedure is 
thought to be too burdensome, and in such cases the selection of students 
will be made by using their dates of birth. Thus for a school which had 
two tickets in the draw, so that half its students are to be included in 
the sample, those would be included whose birthdays occurred on an even 
(or alternatively of course an odd) day of the month. It is clear that 
under this procedure there would be very few cases where the number with 
odd birthdays was markedly different from the number with even birthdays, 
but when such a case did occur the school would be asked to choose the 
larger half and discard from it by selecting for discard birthdays spread 
evenly round the year. Since the student's date of birth is a matter of 
record, and since students will be required to enter it on a question- 
naire, this should be a safe method of selection. 
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It may be noted that the selection of students from selected 
schools has been greatly simplified by the decision to postpone the 
testing of some subjects to a later stage* This has reduced the num- 
ber of subjects to be tested at the present stage to a point at which 
it is reasonable to ask all selected students to take all relevant 
tests* Consequently the elaborate arrangements devised, under the head- 
ing of packing and pairing, to avoid placing too great a burden on any 

student. 



It was noted above that three countries have not adopted the 
general sampling plan without modification* These are the United States, 
India, and France* Because of the immense size of the country, the 
United States sampling is to be carried out in three stages. The first 
stage will be the selection of areas. Within selected areas schools 
w iH be selected, and within selected schools, students* The advantage 
of this plan, in a country so large, is that the more detailed frames 
for the selection of schools need only be constructed within the se- 
lected areas. The design effects will, of course, be somewhat greater 
than they would be if the same number of schools were selected as the 
first stage, but this will be offset by selecting a rather larger num- 
ber of schools* 

There is an additional difficulty in the Indian case. Not only 
is the country of immense size, but the principal language varies from 
one part to another* Consequently it has been decided to limit the 
survey to the six states where Hindi is the principal spoken language. 
Within these states there is already in existence a master sample of 
communities, and this master sample will be used as the frame for draw- 
ing the sample of schools* There is a great deal to be said for both 
these decisions. To extend the survey to cover other states and lan- 
guages would strain the resources available to the breaking point* So 
would a decision to ignore the master sample and construct a new frame 
from the ground. It is true that the standard errors calculated in 
processing the data will relate to fluctuations from the master sample, 
and not from the complete population of the six states, but it seems 
reasonable to hold that this will not introduce much additional uncer- 
tainty, because India, under the guidance of Dr. Mahalonobis whose repu- 
tation is international, has been the home of area sampling. 

The remaining exception is France • The French scheme follows 
the ordinary pattern in the first stage, with the selection of schools 
from a stratified frame using measures of size. 3ut at the second stage 
the selection within selected schools is made not by selecting students 
but by selecting classes. This is because the French authorities object 
to a cl as s being broken up by including some of its members, but not 
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others, in the survey. It should be noted that in France the survey 
will only cover Population iv,and that the classes with which we are 
concerned contain the whole of Population IV and nothing but Population IV 
Furthermore it is possible to grade the classes constituting Population IV 
within each school. Were it not for this fact the correlations emerging 
from the survey would be greatly attenuated, since cases will arise when 
a school with good teachers, good staffing ratio, and excellent facili- 
ties is represented by its worst class, and vice versa. But by using 
the class grade as an unsealed, or categorised, variable in the regres- 
sion this attenuation can be avoided. 



Although the topic does not strictly fall under the head of samp- 
ling design a cross reference should perhaps be made at this point to 
the plans, set out elsewhere, for taking account of the difference, where 
it exists, between the population in an age group and the population in 
school. This applies to the other populations as well. It is of course 
a topic of prime importance, but the plans for dealing with it have been 
set out elsewhere. 



9.2 £ 2 ®! 2 ®©sJBetween Countries 

The questions we shall seek to answer fall under two heads: 



1, Does the evidence support the view that the underlying causal 
systems are generally similar in all the participating countries? Or do 
the countries fall into groups, with the countries in each group resemb- 
ling one another more closely than they resemble countries in other groups? 
And in either case what are the main features? 



2, Is the level of achievement, at each stage studied, much the 
same in all participating countries, when due allowance is made for the 
proportion of the population in school, and for the length of schooling, 
or is there a wide variation between countries, even after these allow- 
ances have been made? And how does the new evidence compare with that 
from our mathematics study, where, for example, the very wide range 
shown initially between terminal students in different countries was 
greatly reduced when the allowances indicated were made? 



: f 



3* There is a prior question which might be regarded as falling 
under the head of analysis between countries. Are the measures of achieve- 
ment equally appropriate for all the participating countries? In devising 
their tests the subject committees have endeavoured to cover all the 
ground that is thought to be of major importance in any of the partici- 
pating countries. If they have been successful in this the question of 
fairness between countries resolves itself into the question of whether 
the most appropriate weight has been attached to the different parts of 
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the tests* The part scores have assigned weights, given by the subject 
committees in their allocation of items to the different parts, and at 
an early stage in the data processing a componentanalysiswill b 
ried out to see whether these weights need modification. If ao, the 
modified weights determined by the component analysis will be used to 
>*ftAcope the measures for all subsequent analyses. In the mathematics 
^udv no re-scoring wa^ needed , since the correlations between the scores 
^th the assigned weights and those with the modified weights were ex- 

tremely high* 

The answers to the first question above will be sought by com- 
-arine the regression analyses for the different countries. From our 
earlier mathematics survey it seems likely that some of our groups of 
or edic tor* variable s will have effects of roughly the same magnitude, 
at any rate in those countries which can be thought of as sharing a 
common culture. Other groups of predictor yariables may emerg 

notably different effects, in different cultural g«>“Ps -SSrt^Sict. 
and vet other croups of variables may turn out to have different effects 

even^wi thin* the saL cultural group of ' ^f^sti^es'or 

oast experience alike give grounds for expecting that the estim t 
^upl^f variables will be considerably more stable than those for 
single variables. In forming the groups of variables we shall Je guided 
first dace by their nature; for example we shall group together 
iLse v^iable^having to do with parents, and those with teachers. In 
4 .* j. ^ affect of a group of variables we shall use more than on 

technique? For example, we shall sum the regression 
each group, and sum the contributions when each triable “ 

? 7i g rrrrnfTninn but we shall also find the sum of the contribution 
of fp r w group when it is taken in what appears to be its most appropria e 
place in 8 the P order. At the outset there seems to be a f ^rly strong 
?nifial probability that the different techniques will all lead to the 

Hr Sar AMST8.--SSS. 2 ssr 

srsssjffs-r «= s s 

■ .2. « sa.r, : s.’ssss 

. . f • j * j hard to find arguments that are conclusive for 

^re^on coefficients themselves do not depend upon 
the order, nor, of course, do the contributions when each variable is 
t^en last, sit both these modes of statement leave something ^ e ^ lf 
the“otal of the regression variance upon the Joint account of all the 

variables • 
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raa thematics study a common scale was chosen for each vari- 
able. In some cases this scale turned out to be inappropriate for some 
of the participating countries. "Type of teacher training" and "Place 
of parents residence" are examples. To remedy this in the present study 
we intend to make use of categorised or unsealed variables, the categories 
being chosen in each country to suit the actual circumstances. The formal 
identity between multiple regression theory and the theory of least squares 

will then be used to combine the estimates from the scaled and the unsealed 
variables • 

In seeking answers to the second main question our prime effort 
will be to present national mean scores adjusted to take account of dif- 
ferences in the proportion of the population in school at each stage, 
and of the length of the course to the stage. In the mathematics study 
we found that these factors, and particularly the first, accounted for 
more than 60% of the variation in the national means for the terminal 
students, since school attendance was compulsory, both in principle and 
m practice, for the other populations in all the countries that took 
part. But in the present study Chile, India, Iran and Thailand are 
taking part, and in these countries school attendance is far from uni- 
versal at any stage. One way of allowing for this fact will be to give 
percentile scores, the percentiles being reckoned for the whole popula- 

tion, and not merely for that part of the population which is undergoing 
schooling. ° ® 
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In the mathematics study adjustments by regression and by equival- 
en percentiles were made, but the table giving the adjusted scores were 
widely separated from the initial tables of unadjusted means, and there- 

v r f-« attr ^ Cted . leS f attention - ^ is led some misunderstanding. We 
shall wold this risk in the present study by giving the adjusted and 

unadjusted tabies together, giving the former more prominence and explain- 
ing that the unadjusted scores are included merely to show how the ad- 
justed scores were reached. A tentative scheme for the analysis and 
presentation of the IEA evidence is presented in Appendix J. 
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CHAPTER 10 



10 . 0 Xn^roduc^±on 

In any survey involving the administration of tests, question- 
naires and other measuring instruments, the arrangements for getting 
these completed by the chosen sample of students and teachers is clearly 
important. Inevitably there are difficulties in ensuring that the right 
tests, etc., get to the right students and that all understand exactly 
what it is they have to do. In surveys that cross country boundaries, 
especially where many different languages are involved, administrative 
problems are magnified and great care in planning is necessary if errors 
are to be avoided. 



Fortunately in the planning for the present Phase of IEA, the 
experience gained in the mathematics Phase could be drawn upon. In that, 
the administration was considered at three levels and three administra- 
tive manuals, one for each level, were prepared. The three levels were 
(i) the National Centers, that is the National Institutes responsible 
for the administration of the survey in each participating country, (ii) 
the Schools, and (iii) the classes or groups of students actually to re- 
ceive the testing instruments. The administrative procedures employed 
in the mathematics survey and indeed the actual manuals that had been 
prepared were invaluable in guiding the arrangements that had to be made 
for the Phase II survey. 

From the outset, however, it was clear that a number of changes 
■in the administration were going to be necessary. Testing instruments 
in six subject areas instead of one were being prepared and complexities 
were going to be introduced by the fact that there were more countries 
involved and that not every country was intending to test in every sub- 
ject area. Moreover, changes in the methods of obtaining the required 
information from students and teachers— the use of answer cards instead 
of answer sheets—meant that different procedures would need to be em- 
ployed for getting the collected information back to the chosen inter- 
national computing Center. 

The problems associated with the administration of the survey 
were considered from the beginning of Stage 1. Initially, however, much 
attention was given to finding an acceptable compromise between having 
short and perhaps inadequate tests but testing students in every sub- 
ject and having longer and perhaps more efficient tests but only test- 
ing each student in two or at the most three subjects. Ways in which 
students could be adequately "packed and paired" were devised. That is, 
procedures were worked out for sub-sampling the students within any 
given school and arranging them in groups so that the required combina- 
tions of subjects could be achieved with no student taking more than the 
tests in two subject areas. It was, of course, important that the pro- 
cedures so worked out were administratively feasible and moreover accept- 
able by the participating schools. 







10 • 1 Segarat±on_of _S tage_2_and_St ag©^_3 



As work progressed on the construction of the cognitive tests 
in each of the subject areas as well as work on the development of the 
student questionnaires and attitude scales, it became clear that the 
I original ideas on the administration of the survey would have to undergo 

| some change. The fact that amongst the participating countries some 

were only wishing to test in one, two or three subjects and others in 
I four, five or six, meant that there were difficulties in trying to ar- 

i range a common scheme for pairing subjects. However, it was the reali— 

[ zation, as time progressed, that the problems of constructing international 

tests in some of the subject areas were greater than in others, and that 

' in fact, it was unlikely that the tests in three of the areas would have 

| reached the desired stage of completion by the end of 1968, that finally 

I led to a major change in the proposed plans for the administration. 

I By May 1968 it was fairly clear that sufficient progress had 

[ been made in the construction of the tests for Science, Heading Compre- 

l hens ion and Literature that final versions of them would be available 

[ at the latest by November 1968 so that administrative plans could be 

f made for their operational use early in 1969* With the other three sub- 

jects, however, - French, English and Civic Education, not all of the 
construction problems had been solved and the possibility that further 
j pre-testing and analysis work would need to continue to the end of the 

year, if not into 1969, was apparent. The decision was therefore made 
\ to divide the survey stage of the present project into two: Science, 

i Heading Comprehension and Literature would be tested first on one sample 

of students, and the remaining subjects one year later on a separately 
drawn sample. 



This arrangement had certain additional advantages. In the first 
place, it meant that the schemes for "packing and pairing" were unneces- 
sary, thus avoiding the very complex administrative arrangements required 
within schools for matching students and tests. Secondly, this particu- 
lar grouping of subjects had much to commend it. There was a close as- 
sociation between Reading Comprehension and Literature in that they were 
both aspects of one subject - Mother Tongue, and furthermore, this sub- 
ject was in marked contrast with Science, so that, by testing the same 
students in these subjects some evidence might be gained to examine 
hypotheses arising from the concept of the "two cultures." Lastly, every 
country now in IEA had agreed to test in Science — the only subject in 
which such agreement had been achieved. While a few did not want to test 
in Heading Comprehension and rather more wished to exclude Literature, 
this present combination of subjects meant that every country would still 
participate in the new abbreviated Stage 2, even if some did not continue 
with the survey program into Stage 3* 
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10.2 The Need_for_a_Dry_Run 
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In the Mathematics Phase of IEA, 12 countries cooperated. About 
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I’m 000 students and 13,000 teachers from over 5,000 schools produced 
130,000 students mu p, information which had to be processed. 

^ILkTf e^S^at «ch item of this information found its way 
, *w. or school from which it originated to its cor- 

? ^Le for ^ocessing at the computation center, was no mean task. 
toil! so it was P not accomplished without a large number of mistakes oc- 

Ttak 8 ’ m TltL W htSfully 1 tt d thtit ^eduction^” tht present Pha“! 

ssssi. no «*» ; s £ t °; 

arESxfs^S-' 

and moreover, an additional P^ulation Cthe lO studen ts will be 

rSSisiii s: 

transference from its place of origin to the computation center. 

• rt -p rnr^ful nlannin* in the administration is, of course, to see that 
these mishaps do not o!cur, but when new ventures are made into unknown 
territory, this is not always possible. 

It is for this reason that the decision has been 

° ne .^oles^eSi^i^inteiided^in* 1^9°to*conduct^ a^dr^run^for^tage^l. 
In°this^ all procedures, including the administration of the 

SArrj-JSStrs ^ 

Since school years begin at different times in thedifferent 
countries, it is inevitable that the XU survey testing will wt be^ 

ried out everywhere at the same time of the year. since this same 

that testing dates will range from January to Ifevember. Since this sam 
... i i v i. fhe Drv-Run. it was necessary to complete the 

administrativ^plans^wel^before t^’end of 1968 so that those countries 

^hing ^o c^r? out their Dry-Bun early in 1969 would not be delayed. 
Accordingly a document setting out the proposed administrative arrange- 
ments was drawn up in May 1968 and sent out to all National Coders for 
ifco-in initial comment. Once these had been received, the Technical 
"cSto were able to produce first drafts of the Manuals 

which were to contain full details of the administration. 
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10.3 The Three Administrative Manuals 



I 

I 

A 



i 



i 



I 



I' 

I 




? 



1 



o 

ERIC 



•.ILT.LT.mlTLJ 



ii 



HgjUHHgH 



Using the experience gained in Phase I t it was decided to employ 
again the three administrative levels, and to put the details of the ad- 
ministration relevant to these levels into appropriate manuals. Further- 
more, it was agreed that each National Center should appoint a National 
Technical Officer who was to have the general responsibility for carry- 
ing out the survey in his country, and he would have the task of inter- 
preting the meaning of the administrative details included in Manual 1. 

In addition, a person who was to be referred to as the School Coordinator 
would have the responsibility for overseeing the administration in each 
school, and the details of his task would be included in Manual 2. It 
was foreseen that in many cases the School Coordinator would be the Prin- 
cipal of the school, and in o tilers a single person might act as School 
Coordinator for more than one school. Finally, at the end of the chain, 

Manual 3 was prepared for the Test Administrator - the person within the 
school who would have the final responsibility for actually administer- 
ing the tests, questionnaires, etc. 

Manual_l 

This gives the general details of the administrative procedure 
it was expected that most National Centers would follow. (See Vol. II, Booklet 1) 
There is, of course, a built-in flexibility to the IEA administration. 

While it expected for example that all participating countries vdll adminis- 
ter the agreed tests and other instruments, allowance is made for countries 
to ask certain questions in the way that suits them best, and for certain 
"national qpv±ons" which they can give or not as they choose. This same 
flexibility governs the methods used to collect and collate information 
within each country, although for the purposes of Manual 1 certain as- 
sumptions are made. It is assumed, for example, that Answer Cards will 
be used for most student responses - if they are not and a particular 
country intends to use an alternative method, it must work out the de- 
tails of the alternative procedure for itself, provided always that it 
caui get the necessary information back to the IEA processing Center in 
the required form. 

Most of what is included in Manual 1 will apply to the main test- 
ing in 1970, although it will be used initially in the Dry-Run in 1969. 

Certain parts, dealing, for example, with the sampling of schools and 
the printin g of questionnaires will have to be modified before 1970 - 
unl ess, that is, the Dry-Run itself reveals the necessity for making 
changes in the administrative arrangements. 
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Manual 2 



Since this is written for a person who may not previously have 
been aware of IEA, it contains an introduction which briefly explains 
the aims and purposes of the survey. Otherwise it details the respon- 
sibilities of the School Coordinator, and as such the example given in 
Volume II, Booklet 1 may well need modification if particular National 
Centers intend to employ alternative arrangements. 

It was suggested that in order to ensure that the contents of 
Manual 2 are properly understood, meetings of School Coordinators are 
held at which more detailed explanations can be given if required. 



Manual 3 
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As this contains the specific instructions for the administra- 
tion of the particular tests, questionnaires and attitude scales which 
are being administered in each country, there will, of course, be 
variations according to what subjects are being tested. Other varia- 
tions will occur from the example given in Volume II, Booklet 1 if, in 
a particular country the testing is being spread over a different period 
of time, or if additional tests or other instruments are being included 
as "national options." It is suggested that School Coordinators hold 
special briefing meetings for the actual test administrators who it is 
expected in most cases will be teachers. 



10 . 4 The_Develogment_of_the_Adminis trative_Procedures 



While it is possible to set down details of the required adminis- 
tration in writing, the complete understanding of what is intended can 
only be achieved by face to face discussion. Accordingly in September 
1968 a meeting was held of the National Technical Officers from all par- 
ticipating countries. Group and individual sessions were held at which 
all aspects of the administration were discussed, both from the general 
point of view of the efficiency of the survey as a whole, and specifically 
as they were affected by the circumstances pertaining locally in the dif- 
ferent countries. A number of the main procedures were modified as a 
result of this meeting, and subsequently further drafts of Manuals 1 and 2 
were prepared. These in turn, together with Manual 3, were circulated 
to National Centers before the final Council meeting of Stage 1 held at 
the beginning of December. 
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10.5 GENERAL PLANS FOR_C^ENDM_MDJ^ITING_ASSIGNMENTS 

As mentioned in Section 1, the four years from January 1969 to 
December 1972 have been split into two stages - Stage 2 and Stage 3. 
Stage 2 will deal with Science, Reading Comprehension and Literature. 
Stsgd 3 consists of English as a foreign language , French as a foreign 
language and Civic Education. The timetables are as follows :- 





Stage 2 


Stage 3 


1969 s- 

Jan* - Nov* 


Dry run 


Finalisation of 
instruments and 


December 


National Technical 
Officers Meeting 


preparation for 
Dry Run 


1970: 

Jan. - Nov. 


Full Testing 


Dry Run 


June - Dec* 


Analyses 


1971 


Analyses and Write- 
up 


Full Testing 


1972 


Continuation of 
Analyses and Write- 
up 


Analyses and Write 
up 


1973 


Publication of the 


Results 



The dry run is spread across a nine month period. Since the 
final testing must be carried out as near as possible towards the end 
of the school year, the dry run in most cases will take place twelve 
months before this, as it is important for it to be an exact replica, 
only on a small scale, of the full testing. School years end at dif- 
ferent times in different countries and in different hemispheres. While 
the dry run is being carried out in the participating countries, the 
writing and testing of the computer programs will be undertaken centrally. 
The computer programs will be written for Stage 2 in such a way that they 
c an be used for Stage 3 with very little alteration. While the dry run 
for Stage 2 is underway it is hoped to finalise all instruments for 
Stage 3. It is expected that the instruments for French as a foreign 
language will be finalised by March 1969* but it may be that some fur- 
ther small testing will be required in the other two subject areas, 
especially in the attitudinal domains. The results of the dry run for 
Stage 2 should show us the mistakes which have occurred. Procedures 




will be developed for putting right the mistakes and therefore in De- 
cember 1969 or January 1970 it is envisaged that we shall have a meet- 
ing of National Technical Officers in order to go through the new pro- 
cedures so that every National Center is aware of how to correct the 
mistakes made. Also while the dry run is taking place the existing 
hypotheses will be transformed from simple into complex hypotheses and 
decisions will be taken on the sort of variables for matrices and their 
order of entry when carrying out multiple regression analyses. Inter- 
national subject area committees will be involved in the finalisation 
of complex hypotheses and in decisions on their testing. 

Thus it is hoped that in 1970 the full testing can be carried 
out with a minimum of errors. 

It is hoped that the dry run for Stage 3 in 1970 will profit 
from the mistakes made for the Stage 2 dry run, and it may not there- 
fore be necessary to hold a National Technical Officer meeting after 
the Stage 3 dry rim, but this is as yet of course uncertain. 

The bulk of the data processing requiring a large computer will 
be carried out at Columbia University. However, once the inter-corre- 
lation matrices are produced, it is possible that further work based on 
the inter-correlation matrices may be carried out at other centers, and 
in particular at Stockholm, 

The assignments for writing particular sections of the report 
have not yet been made, but certainly the heads of the international 
subject area committees, together with selected members of those com- 
mittees, will work in collaboration with the IEA Council members, mem- 
bers of the IEA central staff and IEA consultants in the writing up of 
the reports. The books will then be published commercially. 
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CHAPTER 11 



THE DEVELOPMENT OF DuTA PROCESSING PROCEDURES 
11.0 Introduction 

The first major problem has been the choice of an adequate com- 
putational facility for our work. The data processing requirements were 
well defined from quite an early date. We would need to handle files of 
data for three or four different age groups in each of about 20 countries. 
Each file would contain about 3»000 student vectors, 1,000 teacher vec- 
tors and 200 school vectors. A typical student vector would contain ap- 
proximately 300 bytes of information, and the teacher and school vectors 
approximately 100 bytes each. The incoming data would be in a variety 
of formats, and its arrival at the data processing center would be spread 
over a period of between three months and 12 months. The analysis to be 
carried out after the organization of the date into sequentially indexed 
files would involve in general sequential processing with heavy amounts 
of computation (test scoring, correlation calculations, etc.). 

It was provisionally decided to design the data processing sys- 
tem for the IBM System 360 computers, since these are available in most 
of the I.E.A. centers of activity, and indeed several member institutes 
of I.E.A. have their own System 3&0 capabilities. This range of machines 
is highly compatible, and in general the programs developed for use on 
one model would run efficiently with only a little modification on other 
models in the range. Particular 360 installations which were investigated 
in some detail included the 360/67 at the University of Newcastle in 
England, the 360/73 at Datacentrallen in Stockholm, the 360/65 at Univer- 
sity College in London, the 360/63 at the IBM Data Centre in London, and 
the 360/92 at the University of Columbia in New York. If possible, the 
analysis will be performed on this last-mentioned machine for the fol- 
lowing reasons: 

(a) It has superior peripheral storage devices and input/output 
facilities to any of the other systems investigated. 

(b) It appears to be the cheapest ($300 to $400 an hour for 
CPU time). 

(c) New York is conveniently located for I.E.A. as several of 
our part-time and consultant personnel are already working 
there. 

The second major problem concerned the form in which data was to 
be transferred from the students who took the tests to the computer at 
the International Center. The possibility of having responses key-punched 
manu ally in each of the participating countries and the resulting Hollerith 
cards shipped to the International Center was rejected on the grounds of 
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expense. Various optical scanning systems for use with marked answer 
sheets or cards were investigated. Of these, the best available appeared 
to be answer cards using the MRC 1501 scanner developed by Lindquist and 
his associates at the University of Iowa. One of these machines had been 
purchased by the Swedish government and was to be installed in Stockholm. 
Investigation showed that adequate time was available on this machine 
and that the cost per student would be two or three times cheaper than 
a system using the comparable IBM 1230. In addition, the answer cards 
had certain very obvious advantages over 1230 answer sheets. They were 
considerably less fragile and could be processed by any regular punched 
card equipment (i.e., sorters, collaters, etc.). It was also possible 
to pre-punch identification codes in the card, wl^ch would enable the 
student and particular test document to be uniquely identified, and 
these punched codes could be read by the optical scanning machine at 
the same time the scanning itself took place. 

Eleven MRC answer cards were designed for use with Stage 2. Two 
of them are for use with Population I, seven are for various uses with 
Populations II, III and IV, one is intended for teachers and one is 
available for unspecified use if a participating institute needs a docu- 
ment on which to collect extra data. These cards are now being printed 
by IBM in Stockholm. 

For certain data it was felt to be better to rely on manually 
key-punched cards for data transfer. This applies to certain question- 
naire and test material for Population I (10 year old students, who may 
experience some difficulty in using answer cards) and to school informa- 
tion which is coded by the school principal in a questionnaire booklet. 
National Centers who doubted the practicability of using answer cards 
in their country were given the alternative of sending in all their data 
on key-punched cards, and at least one country will be exercising the 
option during our dry run activities. As a result of our experiences 
in the Dry Run, this National Center, and perhaps others, may wish to 
change their procedures for the main testing in 1970. 

Below we give in as much detail as is currently available, the 
planned sequence of data processing operations for handling data of one 
particular country. The numbers given in the right-hand column refer 
to the time in months after the actual administration of the tests in 
a particular country. During 1969* the month of administration is likely 
to range from February to October, and in consequence the dates at which 
other stages of the data processing are reached will have a nine-month 
spread. 
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11.1 Timetable 



Step 1 



Months After Date of 
Testing 



National Centers send to the central staff pre- 
cise details of their sample. This consists of: 

(a) the code number for each school 

(b) the number of students to be tested 
in each school 

(c) the number of teachers to be given 
questionnaires in each school 

(d) a list of the tests it is proposed to 
administer 



-2 



Step 2 

Central staff prepare punched MRC cards to 
correspond to these specifications. This will 
probably be done on the N.F.E.R, *s 1130 com- 
puter at Slough in England* The identifica- 
tion codes will be interpreted onto the top 
of each card, and then the cards will be 
sorted into the appropriate order for use by 
the National Center (all cards for one stu- 
dent and then all students for one school 
will appear together in the pack). The cards 
are then sent by the fastest available means 
to the National Center. 

Step 3 

National Centers carry out the administration 
of the testing in the schools which have 
fallen within the sample. 



-1 



Step 4 

National Centers send completed MRC cards to 
Stockholm for scoring. 



+1 
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Months After 

Step 5 Date of Testing 

National Centers send punched card data direct to the 
processing center in New York, retaining a copy of the 
data for themselves to guard against accidental loss 
or damage in transit. 

Step 6 

The MRC cards are read on to tape by the 1301 machine 
in Stockholm. Each card type will go on a separate 
tape. In addition to some checking of the data, the 
items will be re-ordered at this stage into the se- 
quence that we require rather than that in which the 
data from the card is actually read. Duplicate tapes 
are made, one for each card type, and a copy sent to 
New York. +2 

Step 7 

The main data processing begins with the reception of 
the punched cards and magnetic tapes in New York. It 
is proposed that at first each of these be checked on 
the small Teachers College computer to find out what 
each data set contains and whether or not there have 
been any major errors. If errors are disclosed at 
this stage, the data will be referred back to the ap- 
propriate people in Stockholm, Hamburg or the in- 
dividual National Center. 

Step 8 



The incoming data sets, now all on magnetic tape, 
will be processed on the main computer at Columbia 
(360/92). Five files will be created: 

MF: master file containing all international 

student data 

RT: raw teacher data file - this will contain 

all school questionnaire data, together 
with the MRC card type 3 data which contains 
the school ratings of opportunity to learn 
the items in the Science test 




3 



I 



I 



; 

1 



1 

4 

& 

f 

% 

3 




11-4 



$ 

35 

1 






I 

a 



i 










% 













x 











* 



Month After 
Date of Testing 

RS s raw school data file ■ this will contain 
«n school questionnaire data, together 
with the MRC card type 3 data which con- 
tains the school ratings of opportunity 
to learn the items in the Science test 

SS: special Science file containing response 

data for the IVS. Science specialist tests 
which are being given in some countries. 

It is intended that this data will be sub- 
jected to some international analysis to 
answer questions principally concerned 
with curricula 

NO: national option file containing all other 

data collected. It is not intended that 
this file will be included in the inter- 
national analysis, although data will be 
available for national analyses after the 
main international analysis is completed. 

There will be some 60 master files, one for each coun- 
try and each population. Once they are complete, they 
will be stored on magnetic tape (maximum size of file 
estimated at 70,000,000 bytes), but during the crea- 
tion of the files it is anticipated that disk storage 
(2311 or 2314) will be used extensively. The files 
will be organized as follows. Each student will have 
one record which will be the sum of up to 8 input 
records (MRC cards) or 13 input records (punched 
cards). All the students in one school will be stored 
together in ascending sequence of student identifica- 
tion number. Schools will be stored in ascending se- 
quence of school code number. There will be 3 pos- 
sible record lengths for each student within any one 
population, depending upon whether a country has ad- 
ministered (a) General + Science tests, (b) General + 

Science + Reading Comprehension tests, or (c) General + 

Science + Reading Comprehension + Literature tests. 

There are slight variations in record length between 
populations because the test instruments have slightly +4 

different numbers of items. 

Step 9 

While the files are being completed, extensive editing 
and checking will be carried out to ensure that the 
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variables all lie within the correct ranges and 
that the data is essentially complete. Once the 
files are complete, a census will be taken to find 
out exactly how many records there are of each 
type for each stratum in the sample. Reports of 
this census will be sent to National Centers. 

Step 10 



Data containing details of the population and its 
division between the various strata will be com- 
bined with the results of the census to calculate 
sample weights for each student and each school. 

Step 11 



The tests will be scored and certain other vari- 
ables will be derived from combinations of raw 
variables in the master file. 

Step 12 



Two working files (WF1 and WF2) will be produced 
from each master file, i.e., one for each popula- 
tion-country combination; 

WF1 contains school variables, school means of 
teacher variables, and school means of stu- 
dent variables. This will, contain about 
120, 000 bytes for one file (or about 
8,000,000 all told). This could be stored 
with ease on one 2314 disk pack. 

WE2 contains student variables drawn from MF. 
Individual item responses will not be re- 
tained in this file, and are replaced by 
test scores, sub-scores and other derived 
variables (400,000 bytes for one file or 
about 24,000,000 all told). This can be 
stored on one 2314 disk pack. 

Step 13 

Item analysis of all cognitive and attitude scale 
data, including principal components analysis. 



Step 14 



Months After 
Date of Testing 



Computation of marker variables from sample (by 
stratum), then the calculation of univariate sta- 
tistics (means, standard deviations and frequency 
distributions) for all variables using WF1 and WF2. 

These results will be reported in full to National 
Centers • 

Step 15 

Initial screening of variables to decide which of 
them will be included in the main correlation ma- 
trices. The screening will be conducted in the 
following terms: 

(a) the size of the simple correlations with 
the main criterion variables. 

(b) the interest a variable or pair of variables 
may have for testing of specific hypotheses. 



Step 16 

Calculation of bivariate statistics (simple corre- 
lations) for two sets of data, (a) "between" schools 
(WFl) and (b) M within" schools (WF2 - WF1). 

Step 17 

Screening of variables for regression analyses. Vari- 
ables will be chosen from the correlation matrix for 
inclusion in regression analyses in terms of their 
intrinsic interest and the size of their simple corre- 
lations with one or more of the criteria. The vari- 
ables so selected will be grouped together according 
to the type of variables (e. g. , home background). 



Step 18 

Regression analysis. This will be carried out to pre- 
dict the various criterion scores from the other 
variables. The variables will enter the regression 
equations one group at a time and estimates of group 
contribution to the explained variance will be made 
in addition to calculating estimates of the individ- 
ual contributions of variables. 




Step 19 



Mriths After 
Date of Testing 



Computer analysis required to test hypotheses which 
were not covered by the regression analysis. 

Step 20 

Regression analysis and other hypothesis test results 
are reported to National Centers for their considera- 
tion. Any necessary re-runs of these tests are car- 
ried out. +7 

Step 21 

Where necessary national data files will be recon- 
stituted to include national option data. The 
files will be put into a format acceptable to the 
National Center and the data will be returned for 
national processing. 

Step 22 

Conclusion of main data processing for one country. +9 



The concluding stage will be reached between December 1969 and 
August 1970 for the Dry Run, depending upon the country with which we 
are concerned. As soon as Step 12 has been concluded for a sufficient 
number of countries, a between countries principal components analysis 
will be conducted on the test data to verify the weighting of the dif- 
ferent sub-scores that go to make up the total score. This will cer- 
tainly be needed for Science, and may also be needed for Reading Com- 
prehension and Literature. If necessary, sub-score weighting will be 
revised for the main testing. 

The results of the item analyses, the univariate statistics, 
anv, the comments gathered from National Centers during the Dry Run 
exercise will be used to undertake a revision of the entire test in- 
strument battery by October 13* 1969* A meeting of the National Tech- 
nical Officers will be held in January 1970 to settle any remaining 
problems concerning procedures at National and International levels 0 
Special discussions will be held with those National Centers who are 
planning to test very early in 1970 for whom the discussion at the 
National Technical Officer meeting would be too late. 

The work for the next 12 months, i.e., for the full testing, 
should then follow in the exact sequence outlined above . 



CHAPTER 12 



12.0 POSSIBLE CONSEQUENCES_OF_IEA_STUDLES_OK EDUCATIONAL 
RE S E ARCH~AN D - EDU CAT I ONAL ^POLICY _DE C I S ION S 



The IEA approach to cross national research has been in exist- 
ence for approximately ten years. The essence of this approach is a 
cooperative attack on a series of educational research problems by a 
collection of National Research Centers utilizing a central coordina- 
tion staff, centralized data processing, and expert committees and 
consultants drawn from the participating countries. Although the IEA 
studies to date have been based on the use of tests and questionnaires 
and quantitative methods, it is clear that IEA research procedures 
need not be limited to such methods where the problems being investigated 
can be better studied by other methods. 

Because of the complexity of the problems it has attacked so far, 
the IEA group has been relatively cautious at each step of the research. 

As a result, three to five years have been required for each complete 
study. Hopefully, it will be possible to reduce the time for studies 
in the future while still maintaining the research rigor and cooperation 
of the participating National Research Centers. 

The pilot study of the IEA (Foshay, 1962) was a study of the 
feasibility of doing cross national research with international tests 
and questionnaires. This study demonstrated that international 
instruments could be developed and that, with appropriate care in 
translation and provision for recognizing cultural and national 
differences on specific questions, such instruments could yield comparable 
and meaningful results. The pilot study revealed the great interest of 
the participating National Research Centers in this type of research but 
it also made clear many of the problems of finance, sampling , adminis- 
tration, and interpretation which had to be solved at the national lave • 
In a number of cases, particular National Research Centers were found, 
inadequately staffed to participate in a cooperative cross national type 
of educational research. 

The problems posed in the pilot study were more carefully attacked 
in the Mathematics project. A number of centers could not participate 
because of inadequacies in staffing and finance and they had to be 
replaced by stronger National Research Centers. Flaws in sampling during 
the pilot study had to be corrected by the use of more rigorous sampling 
procedures under the direction of an international sampling expert. 
Detailed procedures for the development of specifications for inter- 
nationally valid tests required far more time and care than had been 
originally anticipated. Empirical tryouts of test material and question- 
naires revealed many flaws in the materials which required modification. 
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One of the consequences of the IEA procedures so far is a clearer 
understanding of the problems of constructing internationally valid 
tests and questionnaires. Procedures involving specialist groups in 
each country, development of national papers, development of inter- 
national specifications followed by careful and imaginative test con- 
struction, criticism at the national level, and empirical validation of 
test material are required for the tests to satisfy both empirical as 
well as content validity for the many national groups involved. We 
believe that in spite of the extraordinary costs in time and money, the 
development of internationally valid research instruments has enormous 
consequences for educational research and educational planning. 

In the Mathematics study ($nd hopefully in the present study) 
these instruments reveal the levels of achievement which have been 
attained by students in various school systems. The diversity between 
national systems as well as within these systems has made it possible to 
secure a fresh look at each system against the background of all the 
other systems. From the national reports so far published it is evident 
that each country is concerned about its present output of mathematical 
competence and each is seeking ways of improving education in this area. 
Most of the countries plan innovative procedures to correct what they 
believe to be inadequacies and look forward to the use of parallel 
mathematics tests at a later stage to determine the effect of the 
educational changes instituted so far. 

Some of the nations have been concerned about dual systems of 
education which attempt to develop a small educational elite in contrast 
with lowered expectations and educational opportunities for the large 
majority of students. The IEA reports, both international and national, 
have highlighted the differential effects of dual systems and comprehensive 
systems of schools. Clearly, further evidence will be necessary before 
massive changes in the educational systems should be instituted. It is 
to be hoped that the current IEA studies will shed further light on these 
problems and serve to help educational policy makers in the different 
countries make more realistic decisions on the basis of evidence as well 
as national values. 

Since each nation supported its own participation in the IEA, each 
nation regards its portion of the study as being of vital importance to 
its decisions about educational policy. It is this cooperative attack 
on common educational research problems by approximately twenty National 
Research Centers which represents a new dimension in educational 
research. This is in sharp contrast to a highly centralized study where 
the results are regarded as those developed and interpreted by some small 
group of international research experts. Thus, it is believed that the 
IEA finding for a particular country would be reacted to in a defensive 
way if the nation thought of the iresearchers as foreign and without a 
thorough understanding of its own situation. We have found that each 
country accepts its own IEA results as being relatively valid — 
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particular countries. These are basic questions which each nation 
must answer for itself. The IEA can only provide some of the data 
showing the consequences of various decisions in selected countries. 

The IEA has attempted to involve economists, sociologists, 
political scientists, psychologists, and comparative educators in 
planning the data gathering as well as the data analysis. The 
economists, especially, have suggested improved methods of studying 
cost-benefit data for education. It is anticipated that our present 
study will better enable us to understand the ways in which national 
educational characteristics can be investigated in relation to learning 
outcomes. While these are difficult areas to investigate, we believe 
that our present efforts will make a modest improvement over previous 
methods of relating the educational differences among the nations to 
the learning outcomes evaluated by our tests and questionnaires. 

Our inquiry is limited to selected subjects and to selected 
countries. However, our larger goal has to do with an improved under- 
standing of the factors which account for educational outcomes at the 
student and school level. This is a long-term inquiry and we hope 
that each IEA study will yield some gains in our progress toward a 
fuller understanding of education and how to improve it. 
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PROPOSALS TO IEA FOR FUTURE WORK 





APPENDIX A-l 



Memorandum 



Proposal for an International 
Investigation of Instruction in Civics 

Prepared by 

B. Bromsjo and Committee 
Stockholm 



1 

! 



o 

ERIC 



Starting-point 



When education authorities introduce a new subject, they do so with de- 
finite ends in view. The intention is to achieve results in relation 
to the objectives set up for the subject. A certain number of teaching 
periods is allotted to the new subject, but the results achieved depend 
upon many so-called intervening variables, which must be taken into 
consideration when the results of instruction are to be judged. The 
model shows some important factors connected with the subject civics. 
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The intervening variable, which may be most important for the success 
of the subject, is instruction. This may be divided into methods of 
instruction and subject matter. But this variable, about which the 
authorities give advice and instruction, is in its turn affected by 
teaching aids, including text books, the teacher (training and "philos- 
ophy") and the tradition which may be formed in certain cases. (See 
Figure 1) 

Some spare-time activities may also be regarded as results of the 
work of the school. These may, in their turn, influence the result of 
instruction in several ways. 



The school endeavours in the last analysis to achieve long-range 
results, called in the model ability to satisfy the demands set up by 
society. 

Proposal for an investigation 

It would naturally be of prime interest if the projec$:ed IEA 
investigation made ability to satisfy demands set up by society its 
central task. But in this we are faced with great problems, arising 
out of lack of evaluation instruments. It will also be difficult to 
keep the variable called spare-time activities under control, since 
such activities are not purely the result of school work. Hence it 
seems necessary to limit the study to the variables which can be 
assigned more directly to the instruction provided by school. 

The result variables suggested as subjects of evaluation are: 



A. Attitudes Attitude to minority groups, civil phenomena, the 

school, religion etc. 

B. Interests Different school subjects, various aspects of 

civics, different forms of work. 

C. Critical thinking Evaluation of various kinds of information* 

Reasons for holding certain opinions etc. 

D. Insight Causal connection. Importance of geographical and 

biological conditions. Historical aspects (e.g. 
economic problems, different kinds of intervention 
by society, development of society). 

E. Knowledge The course studied. 

The parts of the subject which may be of interest in this connec- 
tion are: 

1. Problems of co-existence (The function of the small group.) 

2. Purpose of society. Rights and duties. 

3. Working life. Occupations and choice of occupation. 

4. Social problems. Social policy. 

5. Law and justice. 

6. Economic problems. 

7. Democracy and problems of democracy (including propaganda, 
political parties, organizations, formation of opinion). 

8. International problems. 
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In accordance with this, the investigation model would be as below 
in respect of the result variables: 
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The investigation would refer to pupils aged about 

I 13 years 

II 16 years 

III 19 years 

Investigation instruments: Tests. Interests and attitude inventories. 
Possible types of items are given in the following pages. 

The general plan of the investigation ahould allow of the special 
treatment of differences in attitudes, interests, knowledge etc. between 
the sexes. 

It should be further observed, possibly in connection with sane 
other subject being investigated, to what extent the instruction given 
at school brings the pupils into contact with material in which other 
countries and people are described as curiosities, that is to say, 
strange customs and traditions are stressed without any background or 
explanation being given. 

Since it is then of interest, naturally, to give the background of 
the results of instruction, it seems urgent to collect data referring to 
this background. It is therefore suggested that the teachers of classes 
affected by the investigation shall answer a questionnaire. 

The teachers may be asked about 

1. The objectives of instruction. 

2. Methods of work applied. 

3. Tuition time devoted to the various items. 

4. Conditions of work (material needed, views on material for 
different kinds of pupils, textbooks etc.). 

5. Their own training and other background factors. 

Some of the types of items which could be included in the interna- 
tional investigation in civics are suggested in the following material. 
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Pupils interest in different subjects 



Below are s number of school subjects, and after them are squares 
in columns. Above the columns are the headings "Very interesting", 
"Rather interesting" and so on. Make crosses in the squares to show 
what you think of the subjects. If you think a subject is neither 
interesting nor dull, put a cross in a square under the heading "Neither 
interesting nor dull". Show what you think about all the subjects you 
have at school. 



Neither 





Very 


Rather 


interesting Rather Very 




interesting 


interesting 


nor dull dull dull 


Mathematics | 1 


P 


p p p 


Civics 


P 


□ 


p p a 


VI 
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a 


p p □ 


VI 


□ 


a 


p p pi 


ff 

etc. 


□ 

Pupils attitude 


□ □ □ Q 

to different methods of work 



(Harnqvist's model) 



Which do you prefer? And which is most usual now? 

Below are different descriptions of work in school, arranged in pairs. 
For each pair you must make a mark (a cross) to show which you yourself 
prefer. You must also show which method is, according to your own 
experience, most usual in school work now. 



I prefer This is most 
this usual now 

That the teacher deals with subject matter in 

the same order as it appears in the textbook 

That the teacher deals with subject matter in 

different order than that in the textbook 

Short questions on facts 

To give an account of a larger section of the 

homework 
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A 3: I, II* 

Eve is very clever at school. Her best subjects are mathematics, 
physics and drawing. She is good in all other subjects, too, except 
needlework, which she does not like, and gymnastics. She is somewhat 
delicate, and her legs are weak. She is diligent, careful and inde- 
pendent in her work, but rather shy and reserved. She is very keen 
on photography in her spare time, does her own developing and printing, 
and mounts the prints herself. Which of the following occupations do 
you think she ought to choose? 

*The code refers to the behavior (A, B, C, etc.), the content (l, 2 — 8) 
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and the population of students (I, II, etc) as shown in the table on page A-13, 
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0 Clerk 
0 Mechanic 
0 Engineer 
0 Child nurse 
0 Chauffeur 

A similar item for male pupils* 

A 4: I. II. Ill 

With which of the following statements do you agree? 
chosen* ) 



(Only one may be 



0 The state should pay all the costs when a person is ill 

0 The state should pay most of the costs when a person is ill 

0 State aid should be dependent upon the income of the person who is 

ill , 

0 The state should pay all the costs for people without money 

0 The state should not interfere in problems of health 

A 5: II 

A sixteen-year-old has been guilty of five thefts of motor-cars, three 
burglaries in shops, and one housebreaking into an empty weekend cottage. 
He has never been charged before* What do you think his sentence should 

be? 

0 Remission of prosecution 
0 Two months' probation 
0 One month's imprisonment 
0 Three months' imprisonment 
0 Twelve months' imprisonment 
A 7:1 

Which of these views is most like yours? 

0 Only girls should help with housework 
0 Girls should help with housework more than boys 
0 Boys and girls should help with housework as much as each other 

0 Boys should help with housework more than girls 

0 Only boys should help with housework 
A 7:1 

Which of these views is most like yours? 

0 Pupils of your age should help with all housework 

0 Pupils of your age should have certain fixed tasks at home; 

e.g* make beds, run errands and the like 
0 Pupils of your age should help with housework when they have 
finished their homework 

0 Pupils of your age should help with housework when they have time 
0 Pupils of your age need not help with housework 

There is a 'newcomer in your class, who is (is from) Where in the 

class do you think this new pupil should sit? 




= occupied place 



Notice your own place, 
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0 A 
0 B 
0 C 
0 D 
0 E 

C 1:111 

Catchwords and slogans are often used— in politics, for example. They 
are hardly ever wholly correct or true. One of the statements below 
is quite false. Which? 



0 Politics is the art of doing what is possible 
0 The world is a powder-magazine which may explode at any moment 
0 Propaganda is necessary 

0 As long as people negotiate they do not fight 
0 The World War brought peace to the world 

C 2: III 

If the highway system is the responsibility of the state, it does not 
follow that the state should 

0 Issue general traffic regulations 

0 Acquire money for maintenance 

0 Manufacture vehicles 

0 Punish those who violate traffic regulations 
0 Close highways when necessary 



C 5: II. Ill 

Juvenile delinquency has increased greatly since the end of the war, 
particularly in towns. What conclusion can be drawn from this? 



0 Peace led to increased crime 

0 Young people in towns are "worse" than young people in the country 
0 The police are more efficient in the towns than in the country 
0 Young people are "worse" now than formerly 
0 None of the above conclusions can be drawn 




C 6: III 

Since the end of World War II (1945) , prices of many goods have been 
more than doubled. Mark with a cross the statement below which 
supports this claim. 
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0 All people earn more than in 1945 
0 It is setting more difficult to make ends n»c®t 
0 People do not buy as many goods now as they did twenty years ag 
0 Money deposited in banks has sunk in value 

0 The quality of goods is more than twice as high as it was twenty 
years ago. 

One ! of I the^ f ollowing statements is not included in the TO declaration 
of human rights. Which? 

0 Everyone has the right to life, liberty and security of person 

0 Everyone has the right, in the interests of the TO, to ignore 

0 ^onf^shal^be subjected to arbitrary arrest, detention or exile 
0 Ho one may be compelled to belong to an association 
0 Everyone, without discrimination, has the right to equal pay f«-r 

equal work 
C 8: I 

One of the following statements is false. Which? 

0 Women are suitable for engineering work 
0 There are no illiterates in the world now 
0 Negroes are as gifted as whites 
0 India has had Nobel Prize winners 

0 In countries with a high standard of living, children go to 
school for a relatively long time 
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Choose the best answer to the following statement. h if 

Statement : We know that our ambassador to X-land will return home it 

diplomatic relations are broken off with X-land. If we know t 
ambassador to X-land is on his way home we can 

0 Be sure that diplomatic relations with X-land have been broken off 

0 Be sure that diplomatic relations with X-land have not been broken 

0 Not say, from the information we have, whether diplomatic relations 

0 ^s!me e that r i t^ a°ver^ r likely that diplomatic relations with X-land 

0 ^°^r;t br i^ £ fikely that dipUwtic relations with X-land 

have been broken off 

Which 1 of the following pieces of information ought one to obtain first 
about the occupation one is thinking of taking up. 

0 Length of training period 
0 Demands made by the occupation 
0 Level of income 

0 Hours of work 

0 Holidays 
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D 4: I. II 



In most countries, the situation o£ women in the labour market is less 
favourable than that of men. Why? 



0 

0 

0 

0 

0 



Old prejudices remain in the labour market 
Women's performance are inferior to men's 
Women are more interested in housework 
Women are not strong enough 
Men have to support families 



D 6: II. Ill 



Risks of higher prices become greater if people 



0 

0 

0 

0 

0 



Save more 
Buy more 
Import more 
Export more 
Produce more 



D 5: I. II 



What do you think is the main reason for punishing criminals? 



0 

0 

0 

0 

0 



To protect the public from the criminal 

To make the criminal feel ashamed of himself 

To deter other people from committing crime 

To help the criminal to become a good member of society again 

To make the criminal pay his debt to others 



D 7: II 



Below are five statements made by different people, 
said to express a democratic attitude. Which? 



One of them may be 



0 

0 

0 



0 

0 



Children should obey their parents without question 
All people cannot share in the government of a country 
Decisions should be made by ballot after everyone has been allowed 
to express his views 

The state, not the citizen, decides what is best for the country 
The task of employees is to do as they are told, and nothing more 



E 2: II 



What is parliamentarianism? 



0 

0 



0 

0 



The principle that the government is chosen by the party with a 
majority or the greatest influence in parliament 

A democratic system of government with, among other things, universal 
suffrage at parliamentary elections 
That meetings of parliament are public 

The principle that important decisions in the government and parlia- 
ment are preceded by negotiations between the largest parties 



E 6: II. Ill 

A newspaper wrote: "In most western European industrial countries there 
has been gradual inflation ever since the end of the war". What is 
meant by inflation? 
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APPENDIX A- 2 



Testing the Ability in English as a Foreign Language 
A report by the Finnish subcommittee* 



I Principles of Classification 

Language represents a system of communication and the language 
skills can be defined as habits of communication. The messages are 
usually transmitted along auditory and/or visual channels. The trans- 
mission can take place by means of lingual or extralingual signs or cues. 
Phonemes are the basic elements of the communication in the auditory 
channel and graphemes on the visual channel. (In language testing 
only the graphemes represented by letters are usually included.) In 
addition, extralingual material is made use of: the communication 

comprises other visual cues (gestures, expression, acts) as well as 
other auditory cues. 

1 . Perceptual -mot or and meaning components of language 



In the examination of language abilities a distinction should be 
made between jgercej^tua^ and symbolic processes. Perceptual or per- 
ceptual-motor functions represent the lowest level of integration. The 
meanings are, however, essential for a language. The meanings contain 
a symbol relation: they represent some extrapresent object called 

significate (the level f mediating processes) . The concept of meaning 
can be defined as a relationship between a significate and its sign. 
Regardless of the way in which the relationships between the significate 
and the sign are interpreted in detail, the criteria for the recogni- 
tion of the symbol relation should be settled, if the process of 
learning a foreign language is studied since the learning processes 
for perceptual-motor reactions differ from those for the meaningful 
associations. Concept formation is a necessary condition for a 
mediating process. Pure perceptual associations occur, e.g. between 
written and spoke languages, when there is no recognition of symbol 
relation. 

The sounds and their arrangements into larger units with definite 
forms and distributions may be learned without symbol relations by 
young children and, to some extent, by adults, at the early 3tage of 
learning. Usually, however, these elements of the structure of language 
are learned in close connection with meanings. In language testing 
only the “public" meanings (especially the linguistic meanings) which 
are common, constant elements of communication to most members of the 
large community are Assessed through language testing. 

2. Uncoding and decoding 

Both the identification and discrimination of percepts and the 
comprehension of meaningful symbols are acts of recognition or 
"decoding", while the imitation and reproduction of perceptual elements 
and the delivering of meaningful symbols can be regarded as acts of 
production (encoding). These functions are distinguished in a test 
situation: in connection with production the person taking the test 

always reacts by using symbol language (by speaking or writing) , 
whereas recognition which consists of listening or reading processe 



can be indicated by simple extralingual choice reactions. 



Classification scheme for language abilities 



Level of 
integration 
Level 1 



Channel 

Auditory 



Visual 



Level 2 



Auditory 



Recognition 

Identification and dis- 
crimination of sounds 



Production 
Pronunciation, intona- 
tion and stre ss 



Identification and dis- 
criminat io n of let ters 



Recognition Word and 
of words sentence 
patterns 




Spelling, punctuation 
and capitalization 



Active Production of 
vocabulary grammatical 
units 



In the classification scheme, the hierarchy of toe levels of 
integration is determined by the complexity and the number of symbolic 
processes required. In the study of foreign language some categories 
lay be of minor importance, e.g. the identification and recognition 
of letters, provided that the subjects are able to rea • 

Some psychological processes cannot be described within this 
ai-heme The direction of the associations between the native and 
theToUign ‘iSgsrfr oreign-)f oreign , native-foreign , foreign-*, five) 
is imoortant for the study of learning in some categories. The 
interactions between the auditory and visual channel should be consider- 
ed in connection with certain processes. 

In addition to these psychological processes, the structural elements 
of language which constitute the grammar, should be classified. The 
grammatical material of languages can be divided into units: 

(graphemes), morphemes, words, compounds, phrases, clauses, en, 
and sentence sequences constitute these classes at different hierarchial 
levels. These divisions can be made in different ways in different 
languages. The grammatical categories of English language are the 
most natural starting points, when a taxonomy is to b « C “"P^ j 
countries where English is not the native language. At a low level 
of the hierarchy, those habits of articulation, sylla e types, an 
collocation are included which are related to patterns of intonation, 
stress, and rhythm. At higher levels, the patterns of words (words, 
modifications, sequences) are considered. 

4. Associations among percepts and symbols 

Language ability is psychologically based on a ssoc ^tion chains. 

The associations can be divided into two main groups. The first 
comprises associations concerning the overlap between words or 
concepts within one channel of communication. The second comprises 
the corresponding associations among different channels of commun ca 
tion. The P associations based on the structural units of language can 
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be separated from both of these. The similarity relationship only is 
discussed in detail. 



Similarity relationships . Associations indicating the degree of 
similarity may refer to both the perceptual and the meaning level. A 
similarity of meanings implies an equivalence relationship. A partial 
similarity may refer to a subsumption or a partial overlap relationship. 
Rather common in language tests are, e.g. , the following synonymity 
assertions: "Word A has the same meaning as word B. " "Word A has the 

same meaning as its definition B." Relationships indicating the degree 
of similarity can be substituted by a dimension ranging from similar to 
opposite meanings (synonymity - antonymity, + - - ) or by a (half-) 
dimension from similarity to neutrality (+ - 0). If we use the models 
originally introduced by Osgood, in which the word or sentence meanings 
are measured by adjective scales, we may expand them to a more general 
scale, e.g. substantive antonymities, antonyme sentences etc. at the 
extremes of the scale (+ — ) or any words or sentences at the extremes 
of the scale (+ - 0). 



Many other techniques can be used for examining the associations 
which indicate the degree of similarity. In differential psychology it is 
customary to study similarity relationships by means of synonymity 
tests, by word grouping or word analogy tests. The basic sentences 
showing similarity relationships may be transformed into interrogative 
sentences etc. , but they always are included in the instruction of 
language tests irrespective of which techniques are applied for the 
assessment of perceptual or meaning associations. 



5. Problems of learning a foreign language 



The learning tasks presented to the students differ according to 
the nature of their native language. Therefore, cross-language compari- 
sons provide in this case a unique problem for research, going beyond 
the problems concerned with most school subjects. The process of 
learning is closely connected with the amount and the kind of similari - 
ties between the pairs of language (native/foreign) in question. It 
should be possible to apply general principles concerning the effect of 
similarity of material (stimuli and responses) to cross-language compari« 



sons. 



The amount of positive and negative transfer of the habit systems 
can be predicted by systematic linguistic comparisons of the pairs of 
languages. It may not, however, be possible to predict in advance, 
what elements of the language structures and contents are essential 
for the learning of global habits of communication in a new language. 
It should be examined, e.g., to what extent the phonetic similarity 
(variants of one phoneme or different phonemes) is related to the 
distinction between phonemes or to the production of phonemes. The 
same question can be presented with regard to the other elements of 
the language structure and content, e.g, the similarity of intonation, 
the similarity of word sequences or sentence patterns, the amount of 
overlap in vocabulary etc. 
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A linguistic examination of similarities between languages is, 
however, not sufficient for the ourpose of cross- language comparisons. 
Therefore, we should examine whether different languages differ from 
each other in the frequency of the occurence of some structures and 
contents in such cases, when they have similar characteristics. In a 
more general way, an examination of the frequency of presentation of 
various types of learning tasks would be important for the comparison 
of achievements in languages. 

Since the directions of the association chains (native/foreign) 
seem to be very important factors in language ability, the process of 
learning should be examined in detail from this point of view. 

6 . The categories for testing ability in English (as a foreign language ) 

The choice of categories for language tests must depend on the 
nature of the problem studied. For a global assessment of achievement 
it might be sufficient to include a language test with subtests for 
four skills: speaking, listening, writing, and reading. For the 
problems of learning presented above, as well as for more detailed 
diagnostic purposes, it would be necessary to include both global and 
more restricted ("analytical") variables in the battery of language 
tests. 



The following remarks refer to the classification procedure. 

1. Language is regarded as a system of communication. Therefore, 
the definition of grammatical concepts etc. has been omitted, although 
these concepts have been applied to concrete tasks. 

2. Some content areas have been analysed carefully, but these 
classifications have not been included in the report. They would be 
relevant to the comparisons of curricula as well as in test construc- 
tion. 



3. The principles of classification vary to some extent in various 
main categories, because the psychological processes and the technical 
requirements of testing have been considered simultaneously. Ex. the 
"partial production" techniques have been discussed separately. 

4. The system of categories has not been put in the form of 
Bloom's general taxonomy, although many parts of it can be easily 
presented in Bloom's terms. 



II Classification 



1. Pronunciation 



Identification and production of vowels and consonants in different 
contexts. 

Recognition of sound segments and discrimination between them without 
recourse to any specific meaning . 

The sounds to be recognized are usually presented as parts of words 
and sentences from the tape. 

Ex. : Similar or different? (1) sleep - slip (2) fist - fist? (3) jeep - gyp 
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Recognition of sound segments and discrimination between them through 
comprehension of meaning . 

Ex. : The teacher reads a word or a sentence and the students choose 

from two or more pictures the one that fits the words or sentences. 

Production of sound segments in words 

Ex. : The production of correct sounds can be recorded in a situation 

where the student has to answer questions or to react to other kinds of 
stimuli by using words which contain problems of pronunciation. 

Indirect recording of the production of sound segments. 

This technique is also called "partial production". The student is 
asked if a word that is given to him in writing is the same as the 
sound of another word which is also presented to him in writing. 

Ex. ; Compare the sounds represented by the vowels. Write the numbers 
of the sounds that are the same. 

sea (1) great (2) head (3) cheap (4) breath 

2. Stress (Accentuation) 

To give prominence to some syllables and words over others. Differ* 
ence in prominence may depend on loudness, duration, and pitch. 

Recognition of the syllable receiving a particular stre ss in words. 

Stress cannot be totally distinguished from intonation in these tasks. 

Ex . : The teacher reads aloud a word and the student is asked to mark 

the most prominent syllable (s). 

Production of words with problems of stress . 

The general technique is to ellicit from the student the words or 
sentence containing stress problems. Scoring would be by direct 
observation or by recording. 

Indirect recording of the production . 

The techniques consist of written words or sentences which contain 
stress problems. The student is asked to mark the syllable with the 
primary stress etc. 

Ex. : (1) af-ter (2) with-out (3) al-ways (4) be-fore (5) be-tween 

3. Intonation 



Sequences of pitch in the speech. Normal statements; falling and 
rising tune. 
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Recognition of the intonation without recourse to any specific meaning . 

Ex.: Compare the intonation, not the words, of the three sentences you 

will hear. Write the numbers of sentences that have the same intonation. 

Identification of the intonation through direct meaning of the intona - 
tion . 

There are only some meanings which are closely and consistently related 
to the intonation. 

Ex . : Listen to the intonation. Check the choice that best fits the 

meaning. - Organize an export business? The speaker is probably (1) 
making a report (2) making a request (3) asking for confirmation. 

Production of a correct intonation through meaningful phrases and 
sentences. 

Ex. : Asking questions. The student responds to the questions or 

completes the sentences. 

Indirect production of intonation via writing . 

Ex . : He taught me how to swim. Indicate the intonation. 

Ex. : Similar intonation in two written phrases in the following list: 

(1) a stone house (2) a nice garden (3) a home 

4. Spelling 

Ex . : Select the word that is misspelled in each of the following 

groups of words. (1) auspices (2) cello (3) alternitive (4) proficient 
(5) none wrong 

More co mp lex problems of spelling occur in connection with writing tests. 

5. Punctuation and capitalization . 

The tasks included in the punctuation and capitalization can be 
studied by means of similar techniques. The abilities to be learned 
can be classified as follows: 

Use of period Use of comma 

Use of other punctuation marks . Use of capitals in names 

Use of capitals to indicate gramma tical r elationships 

Use of capitals to indicate emotional tone 

Other uses of capitals. 

Since there are great similarities among different languages in 
the use of punctuation and capitalization, all the uses described above 
can be studied within a single test of recognition or production. 
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Recognition of errors in punctuation and capitalization 

I . I 

[ Ex. : They climbed into an aeroplane. Soon the aeroplane was high in the 

1 aky. j 

| | 

| (1) soon (2) Soon I 

j* I 

I I 

Production of correct punctuation and capitalization 

| ( 

I Ex. : Do you plan to come tomorrow ( ) Yes, I do. 

(1) (2) . (3) ? (4) : 

I 

[ 6. Vocabulary f 

! p 

| In the study of vocabulary the words are dealt with as lexical % 

units, not as grammatical units. In the selection of words the similar!- • 

t ties between the forms, meanings, and distributions between the native | 

and foreign language should be examined separately. jj 

i j 

Recognition of words, oral presentation j 

I i 

Ex . : Meaning of integrity : j 

j (1) intelligence (2) uprightness (3) intrigue (4) weakness 



i 

: 



Recognition of words, written presentation 

Ex. : He wears glasses (1) for the rain (2) to see with (3) for ice j 

(4) to drink with j 

I 

I 

Translation as a recognition test j 

{ 

1 

The alternatives are given via the native language. j 

1 

] 

Production of words 

I 

The meaning can be provided in different ways: in foreign language 

context, in picture context, and in the native language context. | 

| 

Ex . : What do you see in the picture? | 

Indirect measures for the production of words 

j 

Ex. : I admire him and wish to - banquet as a guest. { 

' I 

(1) a t (2) s d (3) i . k j 

j 

7. Grammatical structure 
■ ■ 1 ■ " 

I 

i 

The patterns of arrangement of parts of words into words and the j 

patterns of arrangement of words in sentences. The testing of grammar 
cannot take place by using similarity comparisons in the same way as in 
the previous parts of the language ability. The elements of the j 



o 

ERIC 



A-16 



t 



















gramnatical ability are grammatical patterns, patterns of words and 
sentences. An arrangement of words that has a meaning over the separate 
j meanings of the words that constitute it, and is the model for parts of 

which other words can be substituted without changing the meaning of the 
arrangement, constitutes a grammatical pattern. The division of grammar 
into learning tasks is reported in a separate report. 

The measures used for an assessment of grammatical ability can be 
classified as follows: 

i 

\ Knowledge of isolated patterns 



j Ex. : Write the possessive plural of these words: 

(1) woman - (2) donkey - 

'4 

Recognition of word patterns within sentences 

Ex. : (1) Boys (2) The boy (3) I (4) We - strikes the car and runs. 

i: 

l Ex. : (1) Who (2) Whom do you suppose it was? 

I 

! 

Recognition of phrases and sentence patterns 

Ex.: (1) Although (2) Inspite of being (3) Although I was - barely five, 

my father sent me to school that fall. 

Production of word patterns within sentences 

Ex . : I received a letter (write) - in English yesterday. 

Partial production of patterns 

Ex. : Rearranging parts of a scrambled sentence. 



I will send - (1) as soon as (2) you (3) a telegram (4) to London 
(5) I get 

8. Comprehension of meanings in phrases and sentences . 



Comprehension consists of all the elements of language which have 
been examined in the previous sections. It is a more complex task in 
which an integration of several aspects of recognition is required. 
Comprehension is examined as a response to both auditory and visual 
stimuli (listening, reading). The material presented is usually 
similar except for some tasks: Tables, graphs, etc. material can be inclu 

ded and scanning tasks can be presented in the reading comprehension 

I tests. 

t 

T 

j In addition to the integration of elementary functions, some other 

I aspects of language achievement can be assessed by means of complex 

tests. Among them the speed of comprehension (or the speed of stimulus 

I 
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administration) may be of particular importance 
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Selection and grouping of ideas to form an overall pattern of thought, 
skimming of material. 

Following directions Drawing inferences Reference skills 

Organization of parts of a book, reading graphs, alphabetizing, 
use of index, reading a map, etc. 

Some skills which are sometimes included in comprehension were 
dealt with in the section on vocabulary. 

The tests for auditory and reading comprehension are usually con- 
structed according to the following principle: 

the student has to answer questions to indicate whether he has compre- 
hended the content played from the tape or the content he has read. 

9. Speaking and writing 

Auditory stimuli and auditory responses 

Ex . : The native speaker asks questions which are recorded on the tape. 

Immediately after each question another voice from the tape tells the 
student in his native language what to reply. 

Ex . : Description of pictures 

Writing by completion 

Ex . : The student is expected to complete incomplete sentences 

Writing compositions 

Style 

Ratings based on written compositions 
Partial production of written performance 

Ex . : Complete the blanks from, the context or the translations then check 

the alternative that fits your answer. 

Translation tests are a particular class of writing tests. The 
production of correct answers is determined by the associations between 
the native language and English. In these tests the direction of the 
association chain is a very important factor. Therefore, two types 
of translation tests should be constructed: one from the native 

language to English, and one from English to the native language. 

Very few examples of the comprehension and writing tests are 
presented here, because these complex tests are included in all the 
test batteries of language achievement. 
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XII Some Problems and Procedu res Suggested 
1. Theoretical approach 

From the point of view of learning theory the study of foreign 
languages provides new problems in addition to these which are common 
to most school subjects, since the learning situation is different 
for the students depending on their native language. These problems 
should be emphasized in the plan for the cross -national study. The 
most important explanatory concept may be the degree of similarity 
between the native language and English. 

For this purpose accurate comparisons between pairs of lai^guages 
should be carried out at the pre-test stage by means of both linguisti c 
and educational research. Theoretically it is possible to predict a_ 
great deal of the differences in the early achievement in English, if 
we know how great are the distances between the native language and En- 
glish in the psychological dimension of similarity. On the other hand, 
we may not be able to predict in advance what similarities between 
the pairs of languages are most important, especially at the early 
stages of learning. Furthermore, the similarity between some 
structural elements may disturb learning, if the discrimination of 
small differences is too difficult. 



There is, for instance, information available concerning the 
similarity of the systems of phonemes between various languages. In 
addition to this information, children can be examined to show y 
discriminations they are able to make in perception as well as in 
vocal expression. If these comparisons are carried out ac successive 
8 t ages of learning, very important conclusions might be made. 

Scaling techniques should be used at the pre-test stage for this 
purpose. The number of comparisons will not be too high, if only the 
similarity between English and each native language is studied. An 
overall estimation of similarity is not useful. The task should be 
restricted and defined by selecting some aspects of language for close 

observation, e.g.. 



- phonemes - words -sentence patterns or grammatical structures 

Accordingly, the similarity (or the amount of overlap) between 
English and each native language is rated separately for these aspects, 
including in the comparison such elements of content and grammar which 
are usually taught during the first three years of study. Hypotheses 
should be made on the basis of these similarity scalings. 



i 



It is important that the languages to be studied vary greatly in 
their similarity with English. The list of languages chosen for this 
part of the study could be approximately the following: 



| 
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German 

Swedish 

Flemish (Dfjtch) 

French 

Italian or Spanish 

Polish or Serbo-Croatian 

Hindi 

Hebrew 

Japanese 

Finnish 

The £inal choice may depend on how early English is taught in 
schools and what are the populations within each country studying 
English. The main problems can be satisfactorily studied, if at 
least 6-8 different native languages are included. 

The second factor which is essential for any explanation is the 
frequency of repetition of each learning task. The frequency of adminis- 
tering certain stimuli can be used as an indirect measure of the degree 
of learning. Many aspects of teaching methods and procedures can be 
interpreted in terms of the frequency of repetition of some classes 
of stimuli (or tasks). 

This factor cannot be examined very accurately for all the tasks 
to be learned. Some comparisons are, however, possible. Ex.: Some 

countries emphasizes a very limited vocabulary in the beginning of 
learning English so that all the words which occur are repeated 
frequently. The estimation of redundancy in the administration of 
stimuli, based on text books etc., provides an index of the frequency 
of repetition. The direction of associations ( native foreign 
language) may show variations across the countries. Information 
on teaching methods should give an estimate of the strength of 
associations in both directions. Emphasis on spoken vs. written 
language in teaching is also related to the frequency of repetition 
of various kinds of learning tasks. 

In section II (Classification) only such aspects of language 
ability were examined which occur in common batteries of language 
achievement. These types of test may not be sufficient in one respect: 
the amount of generalization of general information or skills should 
also be studied, because different native languages and different 
teaching procedures may differ in this respect. Some aspects of 
adopting general knowledge or skills can be studied by means of 
* artifical languages * . The subjects are required to build grammati- 
cally correct sentences with some artifical or unknown words. 

2. General problems of the crosa-cultural study 

The cross-cultural comparisons of English as a foreign language 
should be carried out within the same general frame as the previous 
part of the I.E.A. project. The background information needed from 
the countries is approximately the same: 

- general values systems adopted in the society 

- general educational policies 
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- educational practices 

teaching methods 
curricula, examinations 
selection and streaming 
teacher qualifications 

- teachers' and students' attitudes (which can be used as both inde- 
pendent and dependent variables in connection with different problems) 

On the basis of experiences gained in the previous enterprise some 
background data should be defined in such a way that the comparability 
of results will increase. 

More detailed information is required in regard to such external 
factors which may increase/decrease the students' readiness to learn 
a foreign language: 

- additional exercises at school: readers, programmed courses etc. 

- second language learned at home 

- other languages taught at school (for advanced students) 

- the effects of mass media 

- general cultural exchange 

The teachers' practices should be examined in great detail. It 
seems to be possible to describe very accurately many aspects of teach- 
ing practices in connection with language learning. 

3. Analysis of curricula 

Some aspects of language ability can be described in very general 
terms only and therefore the curricula cannot be fixed for each grade 
level. Ratings concerning the emphasis given to various aspects in 
English instruction may provide some useful information. 



A list of the early vocabulary in English taught at school seems 
to be important, since the content of items to be presented in problems 
of grammar, etc., depends on the vocabulary which can be expected to 
be conanon to most students. (The average size of vocabulary taught is 
approximately 300-500 words for the first year, 600-1000 for the second 
year, and 900-1500 on the average for the third year of instruction) 

A list of words which occur in some elementary textbooks has been 
prepared and comparisons have been carried out concerning the overlap 
of vocabularies among the textbooks. 

For an accurate comparison of the learning of grammatical struc - 
tures the classification of some standard grammar in English might be 
used. For less detailed studies a selection of sentence patterns might 
provide a sufficient background. Many countries do not have any fixed 
programs in these matters. Lists have been co mpiled by the subcommittee. 
for both types of comparison . 

4. Subjects 

If ability in English is examined at an early stage, only the 
erade level can be chosen as the basis for comparisons (one or two 
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years after the beginning of English studies.) This factor is no 
longer important at the intermediate or terminal level. Results of a 
sample consisting of an age group may be less important than in the 
study of mathematics, since English is a compulsory foreign language 
in very few countries. 

It is suggested by the subcommittee that the hypotheses concerned 
with the process of learning should be examined for a limited sample 
of subjects who have studied English one to three years. (Sample A) 

For an examination of some possible hypotheses it would be important 
to retest some groups at an interval of one year. 

A very detailed test battery should be constructed for Sample A 
and some tests should be administered as individual tests. Depending 
on the problems chosen the size of this sample could be about 200-500 
subjects in each country. 

Another sample (Sample B) corresponding to that used in the 
previous study should be chosen for the overall description of achieve- 
ments. No comparisons have so far been carried out concerning the 
number of English lessons at successive grade levels in various 
countries. Final suggestions for sampling can be made after such 
comparisons. It is considered possible that only the Sample 3 of the 
previous study (pre-universitv level) should be included for these 
purposes. For an examination of hypotheses concerning achievement and 
selectivity, the sampling procedure should, however, be planned in the 
same way as in the previous study. 

5. Test construction 



Tests representing the Level 1 of integration (Classification 
scheme onp.A^llmay be most essential for the part of study which is 
directly related to learning theory. It is suggested that the follow- 
ing types of test are included for the limited sample (Sample A) 

1. Identification and discrimination of sounds 

2. Pronunciation, intonation and stress 

3. Spelling, punctuation and capitalization 

4. Recognition of words 

5. Recognition of sentence patterns 

6. Active vocabulary 

7. Production of grammatical units 

8. Auditory comprehension 

9. Reading comprehension 

10. Speaking 

11. Writing 

Some of these tests can be combined (e.g. 2 and 10, 3 and 11). 



Group tests only can be administered to large samples. Therefore 
various types of 'indirect measures' of production seem to be necessary 
for Sample B. It should be examined, however, whether it 
is possible to collect some material with a direct measure of pronuncia- 
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tion for this Sample B. A written composition test should be included 
in the battery as a direct measure of production. 



The Finnish subcommittee has held two meetings for the discussion 
of the problems of language testing. The following tasks have been 
completed: 

A general examination of curricula in English in certain countries, 
as well as overall comparisons of severs! textbooks. Detailed informa- 
tion for the comparison was obtained especially from Japan and Holland. 

An analysis of the vocabulary included in the textbooks or the 
curricula in the lower grades in some countries. (The tasks was carried 
out by graduate students.) 

An analysis of the sentence patterns taught at lower grades in 
Finnish secondary schools (Mr. Rauno Piirtola). 

An analysis of the structure of grammar taught in Finnish secondary 
schools at lower grades (Mr. Peter Jarrett and others). 

In addition, Mr. Pentti Pitkanen outlined the general classifica- 
tions of the psychological processes and the content matter, collected 
the examples, and drew up the theoretical background. Prof; Mar tti 
Takala participated in the planning of research problems as well as 
in the integration of the tasks. 



Separate reports were prepared on: 

1. List of English words included in the curricula 

2. List of sentence patterns included in the curricula 

3. List for an analysis of grammar 



A-23 




mmmm 



AEPENDIX A-3 



I. E. A. 

POSSIBILITIES OF AN INTERNATIONAL RESEARCH PROJECT IN 
MOTHER TONGUE AT THE SECONDARY SCHOOL LEVEL 

REPORT 



As in the research project on mathematics, the preparatory work is 
organized according to the four following steps: 



1) Brief synthesis of study plans in several countries. 

2) Examination of knowledge and abilities which it would be possible 
to test at the following age levels - 13, 15, and 17, i.e., grades 8, 

10, and 12. 

3) Constructing a table on two axes - one of content and one of 
objectives in the subject matter teaching. 

4) Illustrations of test items corresponding with the cells in the 
table. 



I. Study Plan 



Preliminary comments: 



1) Except for that part which is concerned with knowledge and 
techniques general progress in mother tongue is more qualitative and 
because of this it is not easily analyzable into separate units other 
than simple indices. The successive programs in the matters follow 
the following format: 

a) Beginning new themes (for example, movement from narrative 
to dissertation). 

b) Movement to successive levels of obstruction (for example, 
in reading comprehension). 



II. In various languages, objective evaluation instruments have been 
developed independently according to principles which are not identical 
and often they at different levels. This is the case, for example, with 
base vocabularies. A preparatory comparative inventory of these 
instruments is necessary to be able to see whether it will be possible 
to harmonize the norms. 



III. Especially at the secondary level, the use of the mother tongue 
is intimately connected with intelligent activity on the verbal plane. 

It is a matter of distinguishing that which belongs to the nuanced 
use of the language and abilities in judgment and reasoning. Therefore, 
the application of mental, verbal and non-verbal tests seems to us to 
be indispensable; moreover, the team in charge of the test construction 
should have a knowledge of the mental tests chosen in such a way as to 
avoid confusion. 



IV. Even more than in other subject areas, environmental factors, and 
above all, sociocultural factors have great influence on the nature and 
level of school attainment. Also, it will be necessary to have an index 
system defining the various milieux with a fair amount of precision in 
the student questionnaires. A special subcommittee consisting of 
proefessors of sociology and education should deal with this work. 
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V. Certain areas such as literature and spelling are so different from 
language to language that is hardly likely that it will be possible to 
apply common tests. Nevertheless, it would be interesting to profit 
from the occasion of this research to apply identical tests in these 
areas to samples from countries of the same language. For example, 
Scotland, England, United States, Australia or on the other hand, 

French and Belgium. 

1. In the objective "Education in mother tongue and by (through) 
the mother tongue" is often mentioned. 

Syllabuses 

Documents consulted ; 

England: G.C.E. papers (ordinary advanced levels) 

Scotland: Scottish certificate examination papers (ordinary and higher 

grade) 

France: Time tables and syllabuses of the middle schools (Vuibert, 1961) 

Japan: Mimeographed document in English on the objectives of teaching 

mother tongue in secondary school. 

Germany: Syllabuses from Hessen, Hamburg, Bremen and also the Frankfurt 

tests, 

Netherlands: Syllabuses: 

The presentation and the structure of these documents vary a great 
deal; however, all insist on the final important objective-mastering of 
the language. This is why in the present chapter we will follow the 
functional hierarchy of the subjects rather than their didactic order. 
Functionally, mother tongue is the means of communication par excellence. 
The teaching of it aims at essentially understanding and expression. 

1. All programmes insist on the importance of the spoken language; 
this aspect of study plans will be dealt with first. 

2. It is also necessary for pupils to understand the written word 
of others and to express in writing their own thoughts and feelings. 

3. In each language special relatively stable instruments assure 
social competence in communication; the elements are particularly grammar, 
spelling and vocabulary. These tools differ considerably in relation to 
the historical evolution of the cultures, political regimes and the 
economic conditions of each nation. 

4. Certain objectives of a psychological nature can be found in the 
programmes: 

- the wish to see interest in the mother tongue, or certain aspects 
of it, occupy a high place in the value hierarchy of students; 

- the concern for developing taste and literary sensitivity. 




13 year old Level 

Oral Language 



* 






1 (The syllabus of Hegsen entitles this 'hearing and speaking.') 

\ 

| The European Syllabuses are particularly oriented towards oral 

! summaries of what pupils have read on the one hand and towards reci- 

tation or reading aloud of literary works on the other. 

The Japanese at this level lay a great deal of emphasis on the 
spoken language, particularly participation in a discussion on a given 
theme and training in debating. The Hessian syllabus recommends play- 
acting. 

Written Language 

\ 

% 

| Reading: 1. In continental European countries reading is strictly 

limited to the study of authors; in particular, literary analysis plays 
an important part. The main objective is essential aesthetic training. 

The examples provided draw attention to qualities of Style, to the gen- 
eral tone of the piece, to the structure of the composition and to the 
finer points of detail. Host study plans also recommend reading work 
either at home or at school followed by a written summary or oral or 
written commentaries. 

The suggested texts differ according to country and language: works 
from different epochs in France; modern works except for LaFontaine in the 
French speaking part of Belgium; prose and verse selected from modern 
literature in Holland; stories; ballads and fyric poems in Hamburg. 



2. Independent of literary analysis, Japanese directives lay 
emphasis on the social use of reading at this level and contact with the 
thought and emotion of the author; the following objectives are given: 
adequate use of works of reference, training in summarizing as well as 
the inculcation of a value: the appreciation of fine literature. 

The Belgain (French part) syllabus recommends certain exercises 
in synthesizing: edit a summary or resume, describe the character of a 
person, structure of a story. 



Composition . The syllabuses are similar: 
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- in Belgium (French language): accounts, sketches, descriptions; 

- in Belgium (Flemish language): descriptions, summaries, letters; 

- in France: descriptions, stories; 

- in Germany: accounts, summaries, descriptions (views, men, animals, 

things, machines); 

- in Netherlands: essays and letter writing. 

The Japanese emphasize certain positive objectives: precision, clarity, 
cohesivness in work, keeping to the point. 

In certain countries, specific learning of essay writing is foreseen: 

- in France, establishing a plan and developing a paragraph on a 

set theme. 

- in Belgium (Flemish part) , stylistic exercises aimed at looking 

for the proper term (paronyms, synonyms), division into 
"alineas"; observation of movements, colors, sounds, etc. 
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Linguistic Instruments 



Vocabulary (certain syllabuses combine this with the study of 
j phraseology) . 

| The Belgian syllabus (French) recommends: word structure, paronyms 

and antonyms, distinction between literary and figurative meaning (these 
matters appear two years earlier in the French syllabuses). 

t 

i- 

The Belgian syllabus (Flemish) also contains the exploration of 
clusters of ideas on a particular theme (animals, plants, sports, 
cities, industry • • •) the study of abbreviations in common use. 

In Hessen is emphasized the study of words of foreign origin, roots 
and formation of words, place names, terms in modern vocabulary 
t stemming from the Middle Ages. 



Gratranar (and rnalysis). Most syllabuses deal with the nature, form and 
function of statements as well as the order of words and statements. 
However, there are certain differences: 

-the French syllabus deals specifically with conjunctions, relative 
pronouns and interrogative pronouns; 

-the Belgian syllabus (French) deals with a sense of mood and tone; 
irregular conjugation and punctuation; 

-the Belgian syllabus (Flemish) recommends a detailed study of the 
nature of words and emphasizes the verb and its conjugation. 
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Spelling . The difficulties are specific to each language, 



The French syllabuses mention spelling and more systematically 
grammatical spelling. The Dutch syllabuses emphasize the spelling of 
difficult words and foreign terms. 



15 year old Level 



Oral Language 



The Japanese recommend critical listening to general or literary 
speeches. Although certain European syllabuses 1 mention audio-visual 
aids or attendance at "seances litteraires," they are less explicit 
about the systematic use of teaching methods. 



-All syllabuses mention commentary on reading texts, reading aloud 
and recitation of words which have been subject to literary analysis; 
some mention practice of dialogue or play acting. 



In all syllabuses conversation exercises and discussion on a set 
topic appear. The Belgian syllabus (French) includes presentation of 
themes, personnages, authors and epochs followed by an exchange of 
views. The Japanese suggest initiating pupils to chairing a meeting. 



iHessen mentions criticism of films, television programmes and plays. 
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Reading comprehension. All syllabuses mention resumes or summaries of 
texts or works which have been read. As for literary analysis, the 
choice of works varies a great deal; sometimes they are works from a 
particular epoch, sometimes from given authors and in some cases only 
the type of works to study is given. 



The Scottish Certificate of Education (Ordinary Grade) emphasizes 
answering concrete, simple and specific questions on texts which have 
been read from the fields of drama, poetry, prose and general reading 
in the first four years of secondary education. Questions on prose 
should primarily evaluate understanding of the meaning of words and 
phrases, figurative meanings and the ability to synthesize and evaluate. 



The Japanese directives suggest the reading of reports, speeches 
and criticisms; also the use of reference works, newspapers and 
magazines. They draw particular attention to questions of developing 
themes in a story and the intention of the author. 



Composition. Syllabuses vary: 

-stories, portraits, letter writing, speeches, dialogues, literary 
or moral themes, planning of arguments (France); 

-stories including portraits, amlifications, inter-pupil corres- 
pondence, articles for student magazines (French speaking part of Belium) ; 
-summaries, impressions, discussions (Hamburg); 

-summaries, impressions, opinions (Hessen); 

-letter writing, requests, biographies (Hessen). 

The German and Belgian (French part) syllabuses have detailed 
instructions on stylistics. The French syllabus has precise instructions 
concerning initiation to the art of writing: 

The word and the idea (exaggerations, attenuations, images and 
comparisons) ; 

The phrase, expression of an idea: the affective phrase; the period; 
harmony of the phrase and the choice of form; the development and 
formulation of ideas; study of paragraph construction; intellectual 
probity; sincerity; agreement of thought and form. 



Lineustic instruments 

Vocabulary . The French and Hessian syllabuses recommend the occasional 
study of the history of vocabulary and the evolution of words and their 
meaning with the "explication de texte." 



I 



I 



The Belgian (Flemish) syll&us mentions exercises for improving 
vocabulary: resistance to Gallicisms, provincialisms; improvement of texts 
by introducing variety and precision. The Hamburg syllabus suggests 
word formation and regional forms of vocabulary and language. 



Grammar : Most syllabuses suggest the learning of grammatical notions 

when dealing with "explications d'anteurs" and the correction of essays. 
The French-speaking part of Belgium maintain that these are insufficient 
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and that formal instruction in grammar should be continued at this 
level; it should consist of the revision of difficulties of agreement 
and conjugation; in syntax- -word order, inversion, outline in harmony 
with the intention of the style. The Japanese emphasize an understanding 

of grammar rules. 

Spelling . In general, this is only taught very occasionally at 
this level. However, the Belgian (French part) syllabus lays down the 
continuation of dictations at this level. 

17 year old Level 

Note: The time table of terminal classes in France is cut down by an 
hour a week in Mother tongue; the syllabus is mostly literary and is 
left to the initiative of each teacher. 

Oral Language 

-Presentation and explanation of literary works, development of * 

theme (French speaking part of Belgium). . . f . 

-Expositions in conjunction with essay writing (Flemish speaking 

part of Belgium). , 

-Speeches, spoken criticism of films, television programmes and 

plays (Hessen) • 

The Japanese emphasize the playing of a useful role in meetings and 
they put much stress on students developing the ability to defend their 
opinions clearly (affirm, explain, prove). 

Written Language 

Reading. This is more or less restricted to the explanation of 
works; critical, historic, philosophical in conjunction with courses in 
the history of literature and "literary aesthetics". 

In the Netherlands, mentioning the reading of literary works from 
various periods, mentions in particular, making the students aware of 
the period when they were born. 

In Germany special mention is made of translation of foreign 

authors: in Frence (16 years) the reading of ancient authors is recommend- 
ed in the "modern languages" track (section moderne). 

In Japan the following objectives are outlined: formation of personal 
and critical thought; a deepening of the knowledge of the language; 
development of senstitivity. They suggest dealing with logical and 
philosophical points of view within the area of literary appreciation. 

They also mention literary criticism, the reading of various articles 
culled from reviews. 



A-29 






£*-*K»-iasSK* t— . r 












ttma 



Composition 



Belgian syllabuses, dissertations, critical analysis of contro- 
versial texts, summaries of speeches, free work (French); dissertations 
written portraits of statutes, letter writing, syntheses (Flemish); 
souvenirs, interpretations, summaries of discussions (Hamburg). 

In Japan the long term objective of this teaching is to develop 
logical thought and creativity. 

i 

The Scottish instructions define the requirements of success in 
exams as "a sufficient ability in composition and clear exposition in 
grammatically correct English." 

The ability to tackle more broader topics in successive classes in 
secondary education is stressed in various courses. Hamburg makes 
this quite precise in terms of length of work: from 1 hour at the 
13-year-old level to 3-5 hours at the 17 -year-old- level. In Oliver's 
work on the Joint Matriculation Board it is the ability to tackle 
different domaines which is evaluated: the pupil must develop shortly 
(about 1/2 hour per topic) 3 themes to be chosen from, on the one 
hand, aesthetic questions and political, economic and Philosophical 
questions on the other hand. 

Instruments 
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Vocabulary . This is only dealt with very occasionally at this level. 
However, terminal examinations in Great Britain do take account of this 
aspect of mother tongue either in separate tests (Oliver) or in reading 
comprehension questions (Scottish Certificate of Education) . 

Grammar : In Belgian (French part) the study of questions of syntax 

grammatical commentary on some mode in texts, and the discussion of 
language problems are suggested. 

In Flemish (Belgium) teaching at this level is more oriented 
towards sociological and historical aspects of forms of language. 

In Holland a broad study of grammatical questions is advised. 

In other syllabuses allusion is made more to this is "explication 
de textes litteraries". 

Spelling . Only the Belgian (French part) syllabus mentions the testing 
of spelling by use of dictations. 

Domains to be Explored 



I. Spoken language . 

The objective examination of the 6poken language does not seem 
possible to us within the framework of an international research project. 

*See page 8 in French version. 
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Even when limiting ourselves to countries of the same language and 
taking special sub-samples there are still problems which can only be 
overcome with difficulty: complete equality of physical conditions when 
hearing the sounds, establishing acceptable criteria which overcome the 
diversity of local accents and a sure method of analysis. These are 
beyond the present resources of educational, research. 

An attempt to measure spoken language understanding would be in- 
teresting and would be within our means on condition that in each 
country the television authorities were willing to help by showing 
the same film over the whole country. The subject should, of course, 
be stimulating and capable of comprehension at all levels (this latter 
point should be the object of a preliminary study and of the pre-testing 
of the broadcast). 

The test should be similar to a silent reading test; questions on the 
words spoken, appreciation of the language of the characters and their 
behavior through their language; items attempting to evaluate an under- 
standing of the whole or an important part of the sequence. 

As for the progressive difficulty of items, it should be possible 
to use the same film at all three levels; this should considerably 
diminish the work involved. 



II. Written Language 

A. Reading Comprehension 

This is the domaine where international research is easiest and 
where the terms in the ‘Taxonomy of Educational Objectives* apply most 
completely. There are, however, some special problems to be resolved. 

1. It should be possible to present texts where the m»in effort of 
understanding would have a literary bias. 

2. The choice of passages raises the tricky matter of texts equiva- 
lent in interest and difficulty for all the national samples; preliminary 
work could be undertaken on this by a small committee whose task would 
be to submit suggestions to N.C. and get their reactions to them. 



3. At the secondary level, the problem exists of the distinction 
between a reading comprehension test and a verbal intelligence test. 

To what extent does compulsory reference to the text constitute a 
sufficient criterion? By using a particular method of statistical 
analysis will it be possible to identify these two factors? 

4. Below are some suggestions concerning what might be dealt with 
at the various levels: 

Level 1-13 years 
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Ability to find a particular idea in the text. 

Ability to find positive arguments in the text. 

Ability to find various types of motor or sensory details 

in the text. 
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d) Movement from a text to a symbolic or graphic representation. 

e) Movement from a literal to non-literal sense or vice-versa. 

f) Adequate use of reference works* 

g) Knowledge of current abbreviations. 

Specially for literary texts: 



2 

; 



a) Outline particular qualities of style. 

b) Outline the characteristics of certain details 

c) Description of the psychology of a character. 

Several delicate problems will, however, have to be first of all, 
resolved* 

The proposed theme should be equally stimulating for all "milieux," 
whether it be the course of study (programme) , rural or urban character 
of the regions or the language of the country. As with reading compre- 
hension, a sub-committee could work out a list of themes which could be 
submitted in each country for judgment by n teachers in schools not in 
the sampled schools according to a rating scale technique. 
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If all the subjects were kept, the work load would be prohibitive, 
but it should be possible to bring the work down to an acceptable size | 

by limiting this section to a fraction of all the essays. | 

At each level, several types of themes are dealt with; since ability | 

in essay writing is not a simply entity (a pupil's performances in story- I 

writing, letter writing and dissertations are, far from uniform) it would j 

seem desirable to have two subjects with a short time allowed for each | 

rather than only one subject with a considerable time allowed for it. I 

A global assessment of each essay would be of little interest and | 

would induce very questionable comparisons. On the other hand, useful j 

information may be gained from a standardized analtyic notation with j 

total means at zero — an analogous system to that used in the pilot study; 
the comparison of profiles (or models with n dimensions) thus obtained f 

would render meaningful information; in particular there would be 1 

profile patterns of each linguistic or national group. 

I* 

I 

Furthermore, there would be meaningful information about differentia- f 

ted sections (e.g., sections d'humanities anciennes et modernes) within f 

a national system of education. 

| 

It would be a matter of choosing those tasks appropriate to what is § 

imposed at each level (descriptions, narrations, dissertations, essays . . | 

.); for each of these sub-divisions it is possible to establish a 3, 5, j 

or 7 point rating scale; previous studies have indicated methods of 
quantifying these types of criteria. The reliability of such scoring j 

is satisfactory if the same marker is kept to one question and if his j 

consistency is checked by the retest of a batch of papers. 



This undertaking would not be an isolated case; in the literature 
are a certain number of researchers which may be of use to a work group; 
for example: 
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- appreciation of ideas (clarity and continuity of thought, originality of 
ideas, their interest, vocabulary); appreciation of structure (variety of 
phrases, correction of their structure) (Schonell, 1942); 

- depth (clarity and continuity of thought, originality of ideas, form 
(complexity, variety, correction of form, wealth and precision in 
vocabulary) (Xnstitut Superieur de Pedagogie du Hainaut, 1947); 

- analysis of movement, modes of sensation, imagery, emotion, ideas 
associated with typical profiles (centre de reserche sur 1' expression 
francaise (3) ; 

- readability; Flesch's index - De Landshecre (4) • 

Some types of possible topics, appropriate to particular programmes. 

Level 1-13 years 

a) A short description 

b) A brief account of a personal experience 

c) Summary 

Level 2-15 years 

a) Narration 

b) Portrait 

c) Letter writing 

d) Intervention in a debate 

e) Dialogue 

f) Article or survey for a student magazine. 

Level 3-17 years 

a) Dissertation (long essay) 

b) Lecture 

c) Summary of a lecture 

d) Critical examination of two opposing points of view. 

Note: Other tasks in this context may be suggested; here are some 

examples : 

- The French suggest for the 13 year olds the following: for a particular 
essay title, work out a plan. 

- Complete a sentence freely where the first few words have been given. 
Binet and Marg. Evard used this sort of test in trying to identify 
certain types of inspiration: numerical, observational, imaginatory, a- 
ffective and emotive. 

Certain authors have used this type of question as test items 
awarding one point, a half point, or zero according to given criteria 
on the quality of the answer. We are not in favour of using this sort 
of test: on one hand it is difficult to get reliability internationally; 
on the other hand, the existence of model solutions means that original 
answers are penalized. 



A-33 







mmm 


























III. Instruments 



A. Vocabulary 

Many types of tests have been used to try to assess pupils' 
knowledge of this domain of mother tongue. In order to differentiate 
between recall and recognition, certain authors have used both open- 
ended and multiple choice items; despite the rich documentation on the 
first type, we suggest that it is not worth consulting it because of 
the numerous weaknesses there would be in this context in a large 
international study: length of time required for learning and parti- 
cularly between-language equivalence in the selection of acceptable 
synonyms. 

The types of possible tests are numerous, for example: 

-synonyms and antonyms. 

-choice of work (from among others) to cover a common general idea, 
-abstract word covering several related concrete words. 

-choice of wcrd best completing a given phrase. 

-choice of one word in a series expressing the same idea, only to 
different degrees. 

-from a given group of prefixes or suffixes, choose the one which has 
the common root. 

In an international study the delicate problem is that of construc- 
ting batteries of equal difficulty in all countries. We 11-knowi words in 
one language may be little known in another; thus 'parterre' was one of 
the most difficult in the states on Termann-Merrill' s vocabulary scale 
but was on the 7th rung in the French language. To avoid this it would 
be necessary first of all to construct base vocabularies and then select 
items from the same frequency or difficulty zones in all language groups. 

Level 1-13 years 

a) Knowledge of general rules of composition. 

b) Use of terms (semantemes, pronouns, morphames) 

c) Ability to graduate degrees of same idea 

d) First distinction between literal meaning and figurative meaning 
L evel 2-15 years 

a) Recognition of nuance of meaning. 

b) Recognition of nuance of degree. 

c) Old words and expressions. 

d) Neologisms and their classifical equivalents. 

Level 3-18 years . 

Some of the more difficult preceding items. 
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Grammar 



I 



Most domains in grammar are specific to each language. However, | 

in some aspects it is possible to ask common questions. j 

a) Items appropriate to the nature and function of words and prepositions. 

b) It might be possible to examine the operation of certain rules j 

of punctuation on an international scale; under what conditions would | 

this be possible. r j 

c) It would be possible to have common tests in certain aspects or | 

phraseology- -making a sentence more concise, putting into direct or 

indirect language. | 

But syntax, time, voice, moods of verbs and agreements can only be j 

dealt with within one language group. j 

• «- 1 

With this in mind, here are some suggestions for items on an inter- $ 

national scale. | 



Level 1-13 years 

a) Nature and function of words and propositions. 

b) Rules of punctuation. 

c) Putting into direct or indirect language. 

d) Putting into a more concise form (e.g., form one sentence having the 
same meaning as two other given sentences; eliminate in a text the 
repetition of a noun or verb. 

Level 11-15 years 

a) Correct use of prepositions, subordinate conjunctions and relative and 

interrogative pronouns. 

b) More difficult items from level 7. 

Level III- 17 years . 

The same questions as at Level 2 in terms of measuring the progress made 
during the last two years. 

C. Spelling 

In grammatical spelling, the rules vary completely from one language 
to another and thus a common test would be impossible. 

If spelling scales existed according to the same norms in the differ- 
ent language countries in the research project, a comparative study 
would be possible starting with tests consisting of words from the 
same difficulty zones. But these tests probably do not exist at pre- 
s ent and the construction of them would take several years of a research 

study to itself. 

D. Affective domain - interests and attitudes. 



I 
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In the second volume of their "Taxonomy of Educational Objectives" 
Bloom and his Associates (1) have classified the general affective aims 
in education according to their intensity. They distinguish the follow- 
ing progressive steps: 

-reception 
-active response 

-the evaluating of these things in the minds of the pupils 
-the integration of these values into a hierarchial system 
-the influence of this system on behavior or philosophy in life 

It would be interesting to examine what place the various aspects 
of mother tongue have in the hierarchy of pupils interests and attitudes. 
For sure, the results depend to a large extent on the quality of the 
teachers but the great number of schools will cancel out individual 
differences and would give a reliable enough picture of the whole. 

Various means could be used towards this end: 

a) Preference choice questionnaires between the various aspects studied 
and also subdivisions thereof. 

b) A questionnaire analogous in form to Allport and Vernon's Study 
of Values, 

c) A questionnaire adapted from vocational choice. 

d) An interest test in the form of general knowledge test: from among 

a certain number of world known men, indicate what has made them famous. 

e) On the pretext of editing a journal ask the student to choose from 
x items three successive choices which he would like to keep and three 
others which he would not like to have. 

1) Krathwohl, D. R. ; Bloom, B. S. ; Masia, B.B. Taxonomy of Educational 
Objectives II Affective Domain (New York: David McKay, 1964). 
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Spoken Language 1. Comprehension 


2. Comprehension 

functional aspect 


Written Language 3, Comprehension 

Literary aspect 


4, Expression 


5. Vocabulary 


6. Grammar and Analysis 

Tools 


7, Spelling 
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Conclusions 



I 

l 

International research on outcomes in "Mother Tongue" at the 13, 15 
and 17 year levels is not impossible even if it is not complete. 

\ 

f 

It will hardly be possible to begin to consider (in a comparative 
way) those aspects which differ totally according to the language, in 
| particular literary history and literature, spelling and certain aspects 

i of grammar. 



But work is possible in reading comprehension (and to a less extent 
comprehension of the spoken language - or oral comprehension) composition, 

I analysis, vocabulary (if comparable base vocabularies are used). 

I 

I 

f There is also a unique opportunity to examine other aspects of 

language through common tests for countries using the source language. 

| It would also be opportune to try to estimate, at least at the terminal 

level, attitudes of students towards the Mother Tongue itself and 
f different aspects of it. 

i; 

The work should be different from that in the mathematics phase in 
that the orientation of the research should be even more descriptive 
I than normative: it is profiles or standard test models which should 

be compared rather than raw scores. 

Sr 

I 

I The variety of possible types of tests is considerable in each domain. 

I It should be possible to limit ourselves to 2 or 3 forms but the tests should 

| be so graduated as to allow a comparison of progress from one age to the 

next and, at the same level, from one track to another. The total tests 
I consisting of 4 main blocks could be administered as follows: 13 year 

I old level: A and B; 15 year old level: B and C; 17 year old level: C and D. 



As a result of our considerations, we think that the preparatory 
phase of such a research project would take about 3 years. The next 
phase of IEA should therefore be devoted to another subject area; but, 
in the meantime a committee should be set up consisting of specialists 
in teaching and testing; this committee would then decide on the various 
suggested options presented in this report after consultation with the 
National Centers,, At the same time various sub-committees would carry 
out an analagous work for countries using the source language. 

This undertaking would be long and laborious, but its importance 
is such that the effort would be well worthwhile. 



The committee consisted of: 
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Madame Bowijts 

Mr. De Laridsheere- 

Mr. Jeanfils 

Mr. Rigaux 
Mr. Y. Roger 

Mr. F. Hotyat 



Secretary of the University Consultative Committee 
on Education 

Chef de Travaux (Supervisor) at Liege University 
Teacher at Malimody - In charge of Mother Tongue re- 
search 

Lecturer at Charles Buis Training College, Brussels 
Secretary General of Middle School Dept. , Ministry 
of Education and Culture 
Reporter 
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Proposal for an International Study of Student 
Responses to Literature 



July 28, 1965 



Arthur W. Foshay, Teachers College 

Columbia University 
Alan C. Purves, Educational Testing 

Service, Princeton, 
N.J. 



I. THEORY OF THE PROJECT 



A. The Relation of the Study to Educational Problems. 



We propose to describe how students and teachers interpret and 
evaluate a work of literature, and the interaction between them. The 
study we anticipate here will seek to discern what intellectual opera- 
tions, and patterns of operations, students and teachers use. We seek 
not to measure competence within these patterns, but simply to descr e 
their nature and order; not how well students do, but what they do and 
the order in which they do it. Nor do we seek to measure taste. We 
seek to describe the premises of taste. Later, we propose to describe 
the preferences for various types of literary styles and content, and 
also to compare the preferences of students and teachers. All of tnis 
we propose doing on an international scale. 



We propose to examine these national patterns (for we suppose that 
the patterns will be national in character) among students of differing 
age levels, intellectual abilities, sex, and (where appropriate) educa- 
tional experience, ethnic background, and geographical region. In 
addition, we expect to investigate the patterns of preference of 
teachers in the light of their teaching experience, and amount and kind 

of training. 



In proposing this study, we seek to deal with two interrelated 
problem-areas: the school and schooling on the one hand, certain 

aspects of society on the other. With respect to schools, we hope that 
what follows will make apparent the kind of pedagogical problem we 
hope to clarify through this study. With respect to society and 
culture, some comments will appear in later pages. 



In conducting both of these descriptions, we seek to define the 
nature of the gap between the "literary establishment" and the popular 
mind. That such a gap exists has been a long-standing observation; the 
existance of the gap has greatly complicated both certain aspects of 
public decision making and the development of sound pedagogical practices. 
The teacher has not known whether to impose a set of values, or to work 
with the base that exists in the student's world. In the United States, 
this "schizophrenia" has resulted in the muddled compromise of our 
textbooks which juxtaposes Shakespeare and TV Westerns, treating both 
with equal seriousness and essentially the same intellectual equipment. 
There is, moreover, a gap between the expressed goals of literary critics 
and of teachers of literature, a gap in taste, in objectives, and in 
tasks performed. Such gaps, if left latent and unexamined, eventually 
produce strains of misunderstanding among groups of the population that 
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| lead to difficulties in the making of large-scale decisions, and even, 

i upon occasion, to a kind of social disaffiliation. 

| 

One can describe these gaps by juxtaposing the preferences of the 
three groups--students, teachers, and disciplined critics--and showing 
the nature and degree of the incongruity among them. 

I 

| With respect to educational planning, this study can, by showing 

| gaps, indicate possible bridges. It can show the teacher both the 

l modes of thinking preferred by a literary elite, and those preferred 

by the group he seeks to educate. If the teacher is to be the bridge 
| connecting the elite world and the popular world, he must of course 

| have means for knowing both worlds equally well. As things stand now, 

satisfactory means for knowing how students think about literature are 

| not available. 



i 



B. Relation of critical attitudes and attitudes toward life 



o 

ERIC 



The need for drawing a picture of today's critical temper is im- 
portant in itself. But significant literature is an author's rendering 
of life. It follows, therefore, that the way in which a person is 
taught to approach literature may well be reflected in his approach to 
life. Strength and subtlety in one may well parallel strength and 
subtlety in the other. The intellectual historian can assess the 
climate of opinion of an era by examining its art and its critical 
method and standards. For example, the eighteenth-century standard of ” 
decorum was applied not simply to the work of Boileau, Racine, or Pope, 
but generally to habits of mind, and one may infer the habits of mind 
of the era from the study of these authors. This standard influenced 
attitudes towards religion, government, psychology, warfare, and social 
behavior. Popda Essay on Man is an aesthetic standard made a code of 
life. The reflection of attitudes toward life in attitudes toward art 
and critical postures remains valid in age after age (It is paralleled 
by the reflection of attitudes toward life in attitudes toward rhetoric 
and style, and can form the basis for a similar study of rhetorical 
preferences). The assumption about the relationship of art and life 
underlies the whole field of intellectual history, which often generalizes 
about the values and outlook of a society from its view of art and 
criticism, and, more, from the view of literature held by the critic, 
teacher, or student and the world from which it arises. 



Since literature and criticism are taught, we can see in them at 
least the covert teaching of a view of man, as well as the student's 
reaction. We can see, for instance, the conflict of forces that occurs 
within a student nurtured on a sub-literature— television, radio, and 
popular story magazines— who is exposed to a highly disciplined study 
of great works. That a tension exists between the popular culture 
and the great culture, we know; perhaps we can come to understand 
something of its nature. In different countries, of course, this 
tension exists to a greater or lesser extent, but it is the extent 
which we, in part, seek to determine. 
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c. International implications 
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! The third goal is, of course, the comparative one, for as the modes 

of criticism have differed between eras, so have they differed between 
countries in the same era (as a look at published German and English 
criticism of the late 19th Century would immediately show). One would 
I suspect that although a comparative look at critical preferences might 

f turn up not a single preferred mode but several, the pattern of these 

preferences might differ from nation to nation. 

The exposition of these national patterns will serve the most im- 
I portant function of increasing a national educational system' s awareness 

of similarities and differences between nations. We suspect that cliches 
and popular conceptions contain a modicum of truth: there is, in some 

meaningful sense, a characteristic American, or Englishman, or French- 
man, although there are strong lines of cultural unity among these. It 
is these char ac ter ological differences, just as much as linguistic 
differences, which hamper international understanding and communication. 
These differences are in part differences itt ways of perceiving. One 
may take each experience, like each literary work, as a new one and seek 
to treat it in its own terms. One may, on the other hand, look at each 
work or each experience through a categorical lens, see it in terms of 
tradition, or prescribed classifications. There are analogous 
j differences in ways of interpreting and evaluating a literary experience, 

which parallel ways of interpreting and evaluating any experience. 
Although these differences may be subtle and hard to discern, they are 
vast in import. This study will, in part, depict both differences and 
| similarities. Such a depiction will itself be valuable because it will 

make a particular teacher or nation aware of the existence of other 
modes of perception, interpretation, and evaluation. It is the fostering 
of this sort of awareness that is most important for a group such as IEA. 

II. BACKGROUND 

In making the preliminary investigation, the first step was to write 
a number of critics— men eminent for having discussed the varieties of 
criticism— and from their replies to the question "What are the accepted 
modes of criticism?" to arrive at a working summary of current critical 
modes. To the same end, we also examined a sample of college-freshmen 
and high school essays on a given story, and a sample of teacher res- 
ponses to the question, "How would you teach this story?" 

From these materials and from a subsequent conference of scholars, 
we developed a tentative classification of critical elements (see the 
attached documentary prevues of the elements of criticism and inter- 
pretation). The scheme, arbitrary as it is, accounts for most of the 
intellectual operations of the critical act, and with a slight amount 
of refining can become an operational tool. Yet, how to determine 
what critical apparatus obtains in the work of a given student or 
teacher is a knotty problem, for there are certain imponderables that 
need to be accounted for. 

The first is that different literary genres or individual works may 
call for differing critical approaches and might exclude the student's 
habitual approach. As our preliminary study showed, one work we 
selected called for a symbolic reading, despite the resistance a 
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student or teacher might have for the super imposition of typological 
structures; another called for a social interpretation, again despite 
resistance. 
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The second is that a certain tension exists between the private 
and public reaction to a work of literature. We might say, the modes 
of discourse about a work differ depending on the critic's audience. 

We must consider the very possibility of determining a "true" response. 
It often seems that there is only a rhetorical connection between the 
parts of a student's response; for instance, his interpretation does 
not logically follow from his analysis. 



The third is that since this is a preferential test, the choice 
must be free. We have found that when we offered a list of alternative 
interpretations (orthodox ones by our standards), the responses fell 
into a "taught" and a definable pattern, but that when the students 
had to make their own interpretation, a majority of them gave a reading 
not included in our list— in fact totally at variance with what their 
peers chose. It fell outside of our hypothesis, yet was obviously 
predominant in the minds of students, and, presumably, of teachers. 

Even were we to include more alternatives, we might miss the initial 
response of the reader. Further, we might present him with one that 
had not occurred to him, that would not have followed from his pattern 
of analysis, but that would have charm. 



The final imponderable is how to escape bias as among schools of 
criticism in interpreting the data. We trust that the terms used in 
the classification are neutral, as well as exhaustive. 



III. RELATED RESEARCH 



One may truthfully say that there has been little major research 
in the description of student criticism in the past thirty years. Over 
all studies looms the influential Practical Criticism of I. A. Richards, 
the first attempt to examine objectively the writing of students about 
literature. Richard's work, important to modern criticism as it is, 
deals with a rather limited group (Cambridge undergraduates) , and is 
in no way comparative. The technique, however, the presentation of 
an unknown work with a question leading to a free response, cannot be 
improved upon, and is, in fact the basis for this study. 



A similar study. Experiences d' Analyse textuelle en vue de 1* explica - 
tion litteraire : travaux d'eleves by Servais Etienne (Paris, 1935), 
demonstrates selected students' progress in the genre of the formal 
explication de texte and comments on their successive attempts to relate 
form to content in longer, prepared compositions. In restricting his 
study to a small group of students (aged 17 to 19), M. Etienne is able 
to compare the degrees of proficiency attained by some of them. He is, 
however, chiefly concerned with the excellence rather than with the 
nature of his students' writings. 



Related research in the varieties of modem critical approach is 
also dominated by one work, Theory of Literature by Wallek and Austin 
Warren, which comprised a survey of current critical method. The book 
has been imitated and, to some extent, brought up to date in many 
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histories of contemporary criticism and aesthetics, such as those of 
W.J. Bate, Rene Wellek, R.S. Crane, et.al . , but none of these, naturally 
enough, go beyond the cream of published criticism* 

In the field, research, particularly in England and the United 
States, has been devoted primarily to composition, as Research 
in Written Composition , by Braddock and Lloyd-Jones has shown. Very 
little of this research has dealt with the ways in which students 
react to works of literature, nor has much of that research that has 
been devoted to reading, for the latter has been devoted to the 
cognitive problem of reading. The only study of critical reading that 
comes to mind is John Carroll's "Vectors of Prose Style," which bears 
a minimal relation to the ends of this study. Some good articles have 
emerged from the study of an individual class or an individual teacher's 
work, some good objective tests, such as those of the University of 
Chicago, and the Carnegie Institute of Technology, have evolved; 
little of this bears on this study, because it emanates from a 
critical bias, and it is designed to see whether students have learnt 
a particular task well, not to find out what tasks they have learnt* 

IV. HYPOTHESES OF THE PROJECT 

A. Definitions 



1. A sentence is a (grammatically independent) group of words 
which expresses a distinguishable thought. 

2. A leading sentence is a sentence that expresses an indepen- 
dent whole idea. 

3. A topic sentence is a sentence that expresses an idea con- 
tributed to by one or more other sentences. 

4. A supporting sentence is a sentence which serves to illus- 
trate or expand upon a topic sentence. 

5. An element of criticism or interpretation is a specific 
act of engagement, perception, interpretation, or evaluation of a 
work of literature* 

6. A category is a grouping of elements. In this study 

there are four categories: engagement, perception, interpretation, and 

evaluation. 



7. A pattern of criticism consists of a configuration of 
elements irrespective of category. 

8. Frequency of an element or category is the percentage of 
students using that element or category in a topic or leading sentence, 
regardless of the nunber of sentences devoted to that category 
(whether a student mentions plot once or six times is immaterial). 

9. Intensity of an element or category is the average number 
of sentences devoted to that element or category by a particular popu- 
lation of students. 






B. Assumptions Underlying the Project 



I 



1, A pattern of criticism is manifest in a written comment on 
a specific literary work, 

2, A pattern of criticism may be discerned by analysis of a 
written comment on a literary work, 

3, Patterns of criticism are partial indicia of patterns of 
reacting to, perceiving, interpreting, and evaluating experience, or of 
attitudes toward the world (see Section I of this proposal), 

4, The definition of elements identifies the major approaches 
to a work of literature in the countries of this study, 

5, The patterns of criticism will not be identical with the 
patterns of rhetorical composition (This assumption may be proved by a 
parallel study of rhetoric), 

C, Hypotheses about Nations 

1, Each nation can be identified by prevailing patterns of 
criticism, 

2, The number of dominant patterns that exist for different 
countries will vary, 

D. Hypotheses about Students 

1, The student's pattern of criticism will be a function of 
the educational system he is in, 

2, The student's pattern of criticism within a system will be 
a function of: 

a) age 

b) sex 

c) educational experience 

d) socio-economic status 

e) kind of school 

f) region 

g) teacher training 

3, The student's pattern of criticism will be related to his 
stated literary preferences. 

4, The student's pattern of criticism will be related to that 
of his teacher and to the "official" pattern of established scholars 
and critics, 

5* The array of patterns will broaden as the student becomes 
more advanced. 




A-46 



L 









I 









I 



l 








ft 

l 



t 

s, 



l 




£ 



i 



l 






: 



6. Students will have a smaller array of patterns than 
teachers and teachers than critics. 

E. Hypotheses about Teachers 

1. The teacher* s pattern of criticism will be a function of: 

a) training (highest degree or certificates held and 
principal field of study) 

b) stated reading preferences 

c) length of service 

d) sex 

e) educational and socio-economic status of students 

f) size of school 

g) rural -urban status of school 

h) perception of student's literary potential 

F. Operating Assumptions 

1. A sentence may be categorized according to the element it 

exhibits. 

2. A topic sentence on one element may be supported by 
sentences on another element. (A sentence on character analysis may 

be supported by one or more sentences on narrative line, on diction, or 
on syntactic analysis.) 

G. Operational Hypotheses 

1. The frequency of a given element will differ by country. 

2. The intensity of a given element will differ by country. 

3. Contingent relationships among elements will exist by country. 

4. For a given element in a topic sentence, the elements 

employed in supporting sentences will be the same from country to 
country (e.g. , students, regardless of country, might support a 
character analysis by reference to narrative rather than by 

reference to external moral criteria.)* 

5. The average number of topic sentences in an essay will 

not differ by country. 

6. The average number of leading sentences in an essay will 
not differ by country. 

7. In a country, the frequencies of a given element expressed 
in leading sentences and topic sentences will not differ. 

8. The frequency of a given category will differ by country. 

9. Specific associations of categories will differ by country. 

10. Thd intensity of a given category will differ by country. 

11. The number of categories employed by students will differ 

by country. 



V. PROCEDURE 



I 

i: 

i. 






Since we seek to describe critical responses * we will always keep 
in mind three intentions: the study will not evaluate taste j it will 
not follow a particular critical line; it will not compare achievement. 



l 
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With these in mind, we may consider the following questions, each of 
which names a successive phase of the project: 

1. To what extent are responses to literary works classifiable? 

2. To what extent are preferences for particular responses 
reliable? 

3. What are the modal patterns of response among students and 
teachers in different countries? 

4. In what way can the response preferences be characterized 
as literary responses and as types of Weltanschauungen ? 

The present summary and scoring sheet constitute the first step 
toward developing a system by which we may clarify responses. The 
questions we are now asking of it are: 

1. Is it exhaustive in terms of genres, in terms of response 
media, and in terms of population? 

2. Are the categories mutually exclusive? 

3. Is it reliable, in that it will function identically with 
varying texts and readers? 

The answers to these questions have begun to be determined first 
by polling a larger population of established critics, both in the 
United States and particularly in the other countries involved in the 
study and asking them whether they can think of any additions to the 
summary. The next step was to use the classification in making 
analysis of a variety of responses, and the third was to use it with 
a number of readers to determine inter-reader reliability. We further 
intend to test the system with responses to poetry and prose, with oral 
and written responses, and with both student and teacher populations as 
advanced and slow classes, highly trained and untrained teachers, and 
students of different national origins. This pilot study has been 
accomplished with a small group of students and teachers in the New 
York area, and with other small groups from Belgium and Germany. 

A. Rationale 



The rationale of these pilot investigations is as it will be 
for most of the project. The students are offered a short work (3-5 
pages if a story, under 30 lines if a poem) and are asked to read and 
"discuss" the work. This question enables them to frame a response 
most suitable to them, and enables us to compare that response to the 
expressed teaching preference of the instructor. 

The essay is then read by three readers to assess inter-reader re- 
liability, and to assure accuracy, each reader using a scoring sheet 
which shows the element and type of each sentence that a student writes. 
The scores will then be analyzed for significant patterns in accordance 
with the operational hypotheses above. 

A similar form of reading and analysis will be used on the response 
of teachers to the question, "How would you teach this work to your 
class?" 



The second step is to develop a ’’parsimonious" instrument# We must 
determine: 



a) Whether the responses of a student are reliable within a 
genre and between genres,, 

b) The extent to which a particular literary work determines 
a particular response pattern* 

c) Whether the responses are reliable across oral, written, 
and multiple-choice media* 

To make thase determinations „ we propose to present a student 
population and a teacher population with two works of poetry and two of 
prose, and ask them to respond to one in each genre orally and to one 
in each genre in an essay* Using the classification, we will make a 
content analysis of the responses and determine the reliability of 
the responses. If they prove reliable (which, grossly measured, we 
expect they will) , we can then explore the reliability of a more 
easily scored instrument, checking the responses to this against another 
group of essays and interviews. If this sort of instrument proves un- 
reliable, we will seek to determine the least amount of "free informa- 
tion we will need to determine a response preference. Even with an 
easily scored instrument, we suspect that at every stage of testing 
we will need a modicum of free response material to serve as a check 
and as the source of finer distinctions* 



For the taste-preference section, we will assemble a variety of j 

short tests — typical paragraphs from sub-literature, from best-sellers, | 

from avant garde literature, from academic literature, from "philosophic" 
literature, to name but a few of the array--and present them to teachers 
and students. We will ask them to rank their preferences (both what 

they like and what they think good by stylistic, moral, affective or \ 

whatever criteria) and see what sort of comparability exists between j 

the responses in a population and across two populations. 

During the course of this phase, we will also be continually ! 

testing the literary works to see whether they pre-ordain a certain j 

response. Further, we will seek to determine the sort of literary work 
best suited for a valid response. In making these determinations, we j 

will move to a larger population and seek to determine comparability | 

across sub-groups-- ethnic, cultural, intellectual, sex, grade-level, j 

school- -and between teacher and student responses* From this we will | 

develop a test and analysis procedure that can be operational and effi- j 

cient on an international level. | 

I 

B. Population f 



The population for all phases of this project will consist of 
thirteen year olds and seventeen year olds. The first group is the 
generally comparable one of these children in their last year of 
elementary education; the second that of those who are in the last 
year of secondary education. The sample will, of course, be such 
as to enable analysis according to Hypotheses D and E (above). 
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C. Instrumentation 



1. Critical patterns instrument (two forms) 

a) Derived from content analysis, it will emerge in a 
machine-scoreable form, but will be supplemented by a small number of 
free responses. 

2. Preferred literature instrument (rank order of passages 
by varying criteria). 

3. Inventory of student's attitudes towards the field of 
literature. 

4. Inventory of teacher's attitudes towards students. 

5. Background questionnaire concerning teacher and school. 

6. Background questionnaire concerning students' and teachers' 
educational and home experience of literature. 

As in an exploratory study we have completed, we will present more 
than one work to each student and teacher, and in some cases, more than 
one test. This will again be to determine the reliability of the 
response and provide a clearer view of the patterns. 

Because of the problems of literary translation, and because of the 
cultural differences that might intervene in the response to one literary 
work (the inability of a student from one country to appreciate the 
conflict in a work from a radically different society) we will attempt 
to devise a test that can use an indigenous work but apply to it a set 
of neutral questions; yet we suspect that a better solution would be to 
present a "neutral" work. Then, if a particular nation wishes, it may 
use the form of the test with an indigenous work to arrive at a more 
accurate assessment of its own sub-groups. 

D. Analysis 



From the data gathered, we will determine the modal response 
patterns of a given group or sub-group. We will also determine the 
interrelation of modal response patterns of students and teachers in a 
given educational system. 

The final phase of the project will be the characterization of 
each of the major response patterns. We will do so by returning to the 
classification and inducing from each major pattern its dominant 
features. These features will be examined to see if they are interde- 
pendent and in what ways. From this characterization, we will proceed 
to define the critical attitude it implies and to make a tentative in- 
ference as to the Weltanschauung it implies. 

The characterization will proceed to relate the response patterns 
to the variables gathered in the test (sex, age, etc), but more importantly 

perhaps, to those variables which given some indication of the reading 
or amusement habits and preferences of the various population. From 



A-50 






9 

ERIC 



•.'I.LT.’.TllJ 


















er|c 



this relation, we can infer the interrelation (or lack of commonality) 
between popular "literature" and prescribed literature, between student 
and imposed literary values. This inference will be somewhat tentative, 
'because there are obviously a greater number of variables acting on any 
one person than one can ascertain or even suppose. 



The inferences about Weltanschauunge n will arise from a considera- 
tion of the implications of each pattern. If, for instance, one 
pattern shows a highly formal system of analysis and no evaluation, 
a possible inference would be that a scientific attitude obtain 
all fields of inquiry— an attitude which says that the world exists to 
be examined but not to be Judged, further that man is a logical, but 
perhaps not an evaluating instrument, and that the world exists as 
objective phenomena not to be viewed subjectively. Contrasted to this 
view, might be that presented in a pattern of moral interpretation and 
evaluation with perception limited to the search for allegory. Such 
a pattern would indicate a typological or even anagogical man, one 
who sees things in symbolic terms, who sees all reality in terms o 
paradigms of behavior, as a series of dimunitions from 6ne central 
standard. Obviously, these two views cannot be compared in any 
evaluative fashion, but they can be pointed out. 



We have said that these inferences are tentative, and so they are. 
They are suggestive generalizations based on the ways in which students 
and teacher perceive, describe, and evaluate a given literary o Jec • 



E. Time Schedule 



The perfecting of the instruments would require one year, in 
which time the preliminary data should yield instruments land 2 with 
sufficient cross-reference material derived from essays and interviews 
to insure its reliability. The other instruments can be supplied by that 

time. 



Testing, scoring, and analysis should require two additional years 
with additional time for publication of the results. 



VI. PERSONNEL 



The director of the proposed project would be Prof. Arthur W. Foshay, 
Associate Dean of Research and Field Services, Teachers College, 

Columbia University. The principal technical consultant during the 
developmental phase of the project has been Prof. Alan C. Purves, who 
has left his post as Professor of English at Barnard College, Columbia 
University, to become Professional Associate, Humanities Department, 
Educational Testing Service. It is expected that Dr. Purves will 
continue to consult with the project, which will also continue to have 
access to the other professional services of ETS. 



In addition to the director and general consultant, the project will 
require consultants in literature from several of the participating 
countries, and also a staff of technical and clerical assistants. 
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We assume that the administrative arrangement used in the I.E.A. j 

Mathematics study would continue to serve this project: i.e., a council j 

of participants, a standing committee, and an administrator at Hamburg. ] 

I 

f. 

The Teachers College participant in the study would be the Horace j 

Mann-Lincoln Institute, within which the pilot work of the project has j 

been conducted. It is anticipated that the University of Chicago will 
continue its present cooperative relationship with this study. 

VII. FACILITIES 

Data processing facilities appropriate for the study are available 
at ETS, Columbia University, and the University of Chicago. A 1620 
system is available at Teachers College, Columbia University. Large ^ 
university librarian are available at both Columbia University and the 
University of Chicago. A large technical library is available at ETS. 

VIII. FUNDS 

The proposed study is relatively large in scope and time. While it 
is not possible to present a budget now, the general types of expendi- 
tures can be anticipated: 

1. Direction and travel 

2. Technical assistance 

3. Publishing and communications 

4. Costs within countries 

5. Data processing 

i 

We would propose applying to Project English of the U. S. Office of | 

Education ivv funds. The size of the proposed grant would be in excess 
of $250,000. ] 

I 

IX. PARTICIPANTS 1 

I 

Several general cultural regions are pertinent to a study of 
literature, since these regions have developed distinguishable schools 
of literary criticism within them. Ideally, at least two countries | 

from each of these regions should participate. These regions can J 

be identified by language or by areas, as follows: 

1 

I 

1. French-speaking Europe 

2. Scandinavia j 

3. English-speaking countries 

4. Italo-Hispanic area 

5. Asian countries 

6. German-speaking countries | 

| 
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APPENDIX A- 5 



I. E. A. 

REPORT OF THE USA COMMITTEE ON THE TESTING OF FRENCH 

AS A FOREIGN LANGUAGE 
Members of the Committee 



Professor Wilmarth Starr, Department of Romance Languages, New York 
University 

Mr. F. . Andre Paquette, Modern Language Association and American 
Association of Teachers of French 
Professor John B. Carroll, Harvard University (Chairman) 

Mr. John L.D. Clark, Harvard University 

Introduction 



This is the report of a committee convened at the request of 
Professor S. Bloom of the University of Chicago to outline a provisional 
proposal for an international investigation of student achievement in 
French as a foreign language. The proposal represents strictly a USA 
point of view in the sense that it sets forth possible specifications 
for the investigation as it might appropriately be conducted in the USA. 
It is anticipated that the proposal will be reviewed by an international 
committee and coordinated with proposals received from other participa- 
ting countries in order to develop a uniform design applicable in all 
these countries. 

The teaching of modern foreign languages in the USA 

This section presents a brief summary of the history and current 
status of the teaching of modem foreign languages in the USA, with 
particular attention to French, in order to provide necessary background 
for examining the detailed proposals made as to the design of the study. 

The teaching of modem foreign languages did not becone a normal 
and accepted part of the curriculum in the American secondary school 
until around the middle of the 19th century. At that time, the few 
secondary schools that existed were attended, for the most part, only 
by the relatively small segment of the corresponding age groups in 
the population who intended to go on to some form of higher education. 
Towards the end of the 19th century and continuing into the 20th, the 
evolution of the comprehensive high school brought larger and larger 
sectors of the population into the secondary school. Although total 
foreign language enrollments increased, the proportion of high-school 
students enrolled in foreign language courses tended to decrease, since 
foreign language study was still sought, in the main, by those in 
college preparatory curricula. The pressure to know a foreign language 
for any utilitarian purpose is obviously not as strong in the USA as it 
is in many other countries, e.g. , in those countries in which English 
constitutes a language of wider communication value than the indigenous 
language. On the other hand, the trend in the USA since World War II 
has been a gradual increase in the relative numbers of secondary school 
students electing foreign language study, and for the first time there 
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have been serious and at least partially successful efforts to intro- 
duce foreign language study into the elementary school curriculum, 
even as early as the kindergarten in some cases. These crends are not 
solely due to the fact that increasing numbers of students hope to 
attend college; they also appear to stem from genuine interest in 
foreign language study for its own sake and also for its use in leading 
to better international communication. The NDEA of 1958 gave an 
enormous boost to FL study through its provision of government funds 
for the training of foreign language teachers and for language labora- 
tory equipment in schools. For a general statement on the current 
status of foreign language study in American life, see W.R. Parker's 
The National Interest and Foreign Languages (1957). 

For the first half of the 20th century, the primary aim of foreign 
language teaching in the USA was that of producing a reading knowledge 
of a language. It was believed that such an objective was the only 
reasonable one to expect of students in the relatively short periods 
(usually no more than two years) that students customarily studied 
any one language. There were few teachers who could effectively teach 
the spoken language. The teaching was chiefly of the "gramnar-transla- 
tion" variety; that is, students were taught rules of grammar and 
required to apply them in performing written translation from and into 
the target language and oral translation from the target language. In 
the public mind, the study of foreign language was regarded as generally 
unsuccessful and unproductive. In World War II, however, it was 
dramatically demonstrated that new methods of foreign language teaching, 
giving priority to mastery of spoken language, could be productive and 
successful even in periods of two years of study. The revolution in 
methods of language teaching was so thorough that public attention 
was brought to the desirability of longer periods of language study in 
order to attain true and effective mastery of all aspects of a foreign 
language. Furthermore, methods of language teaching became more re- 
fined and effective, especially with the introduction of the language 
laboratory and other technological aids. Increasing numbers of 
competent foreign language teachers became available, even at elementary 
school levels. Nevertheless, progress in foreign language teaching 
has been uneven. Traditional aims and methods persist and are occasion- 
ally stoutly defended in many schools throughout the country, particularly 
in the less affluent districts. Because the school curriculum is deter- 
mined largely on a local basis, wide variations in the quality of 
instruction are found. 

Although Spanish is the language which is studied by more secondary 
school pupils than anv other language, French runs a close second, and 
in some sections of the country or in many school districts, it is 
elected by more pupils than Spanish or any other language. There is 
still a tendency to regard French as par excellence the language of 
culture and of world communication. German is considerably behind 
French in enrollments, and Russian still trails far behind, even 
though it is increasing in popularity. 
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Specifications for testing student achievement i n French 

Because the study of French can be begun at many different age 
or grade levels - all the way from the kindergarten to the graduate 
school - there is a certain arbitrariness about setting up three levels 
of proficiency to correspond with age levels* Nevertheless, in order 
to accord with the spirit of the suggested guidelines set forth by the 
IEA Conmittee, we recommend that testing be done at three age levels: 
the 13-14 age level (end of grade 8) , the 15-16 age level (end of 
grade 10), and the 17-18 age level (end of grade 12, pre-university). 
However, in the selection of the students to be tested, account should 
be taken of the approximate number of contact hours of instruction they 
have had. Thus, in testing at the end of grade 8, the sample should 
be restricted to those who by that time have had a cumulative total of 
160-180 contact hours of instruction in French. Further details are 
shown in the accompanying chart which indicates the degree of 
competence in each of four basic skills, - listening, speaking, reading, 
and writing, - at each of the three levels. 

No systematic or detailed inventory of either lexicon or syntax 
has been sugeested as the basis for a corpus of testing items, i.e., 
a list of words and constructions that could be divided into three 
parts presuming progress from the simple to the more complex. The 
reason for not doing so is that contemporary language teaching methods 
in the USA increasingly view language learning as a progressive experience 
implying enlargement and enrichment of lexicon together with variations 
upon the basic structure of the spoken language. The irreducible 
elements, therefore, are the complete sound avstem which can be covered 
in a fairly short period of time (and which indeed is automatically 
subsumed as soon as a reasonable number of words are strung together) 
and basic syntax which is implicit in connected rational speech of even 
fairly brief duration. As long as one limits himself to material 
suitable to interest and age level, he is i pso facto including the 
sound system and basic vocabulary and syntax. Progress in language 
learning may be better described, then, by the analogy of concentric 
rings rather than by travel along a linear distance. In the above 
analogy the total sound system and reasonable coverage of syntax 
should be understood as included within the central core of the concen- 
tric rings. It is hoped that the illustrations for the identification 
of levels in the grid will illustrate the above concepts. 

On the other hand, the concept cf frequency of occurrence of 
lexical and syntactical elements in normal language production may 
provide a clue to useful techniques for presenting to the learner items 
of greater frequency first. 

For the matter of lexical inclusion it is suggested that "in two 
high school years or one college year of instruction it is possible to 
inculate an "active" (speaking -writing) vocabulary of between 500 and 
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SKILLS 13-14 age level 15-16 age level 17-18 age level 

end of grade 8 end of grade 10 end of grade 12 (pre-university) 

160-180 contact hours 430-450 contact hours 700-720 contact hours 
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1,000 words, and a "recognition" (hearing -reading) vocabulary of 
approximately 1,500 to 2,500. "Language", of course, is more than a 
body of isolated words that can thus be counted, but these figures 
give us a basis for significant comparison. "1 These figures also seem 
to equate with the basic 1,300 words of Le Francais Fondamental . 

Such a basic vocabulary could be assumed to be covered in about 300 
contact hours, or, in the sequence outlined here, at a point about equal 
to the end of grade 9. 

The problem of syntax is an even more arbitrary one. Useful 
guides are suggested, such as the section "Structures for Four- and Six- 
Year Sequences" in French for Secondary Schools . Bureau of Secondary 
Curriculum Development, New York State Education Department, Albany, 
I960, or the indexes of the more widely used French "grammars" which 
generally cover the basic structures of the language in one volume 
after ordering them in an arbitrary system based upon notions of 
greater to less frequency and simpler to more complex patterns. 
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Sample test materials 

Sample test items for each of the four skills are 
given on pages A-58 through A-7 ?• The approximate level of difficulty 
of each item has been designated, but these are not intended to 
correspond exactly with the three levels at \rtiich the tests are to be 
given, since the tests to be given at each level should include a range 
of difficulty. 

The items are for the most part of the "objective" variety, and in 
most cases they can be scored objectively. They have been drawn from a 
variety of standardized tests now available in the United States. 
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Additional data to be collected from students 

In addition to obtaining student test scores for each of the four 
language skills, the project should undertake, in so far as possible, 
to obtain data on background variables which may be thought of as 
affecting student performance on the tests themselves. It is suggested 
that a detailed questionnaire be prepared for administration to students 
at each of the three proficiency levels at about the same time that the 
tests themselves are being given. The questionnaire would ask the 
student to indicate the nature of his foreign language experiences, both 
in the classroom situation, and in such informal contexts as travel in 
foreign countries, acquaintanceships with foreign visitors, different 
languages spoken in the home, and so forth. 

J-Starr, Wilmarth H. , et. al. Modern Foreign Languages and the 
Academically Talented Student. National Education Association, 1960. p. 79. 

^Gougenheim, G. , Michea, R. , Rivenc, P. , and Sauvageot, A. 

L 1 elaboration du francais elementaire. Paris: Didier, 1956. 
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Although these informal contacts may be expected to contribute to 
student achievement in the foreign language to some extent, it is 
suggested that an analysis of the nature and content of the formal lan- 
guage courses which the student has taken, from the beginning of his 
foreign language study up to the time at which the tests are administered, 
will in the majority of cases show a close correspondence with his 
observed proficiency in the four skill areas. Thus, a major portion of 
the student questionnaire would be devoted to a description by the 
student of the language courses which he has taken. The questionnaire 
would be prepared in such a way as to enable the student to check, 
from a number of descriptive statements about class organization, methods, 
and emphasis, the statements which most closely describe the courses which 
he has had. Examples of items which may be of use in this connection 
include the following: 

1. How many class meetings were there each week? 

one 

two 

three 

four 

f ive or more 

2. How many students were in the class? 

less than 10 

10 to 20 

20 to 30 

more than 30 

3. During the class periods, the students would 

u sually speak in English unless imitating 
the teacher in certain drills 
u sually speak in the foreign language for 
drills, conversation and questions 

4. For this course your class 

a. did not use a language laboratory 

b. used a language laboratory every now and then 

(not regularly every week) 

c. used a language laboratory regularly, - that is, a 

certain amount every week 

If you check "c", answer this question too: 

How much time each week did your class spend in the language 

laboratory? 15 min. 

30 min. 

45 min. 

1 hour 

1% hours 

2 hours 

2% hours 

3 hours 

more than 3 hours 



5. Did native speakers of the foreign language ever come to talk 
to the students during class periods? 
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no, never 

yes, occasionally 

yes, reguar lv 

Experience with questionnaire items of this type indicates that 
quite detailed information about previous language courses may be 
obtained in an economical manner, both from the standpoint of time 
required for administration of the questionnaire and the time required 
for tabulation and analysis of the data obtained. 

In addition to the student's formal and informal language contacts, 
a third variable affecting measured language achievement is student 
motivation for, and general orientation toward, the study of a foreign 
language as such. These aspects can be measured with some validity 
by the inclusion of an appropriate series of items in one part of the 
student questionnaire. Below are some examples of types of items which 
have previously been used in this connection! 

Directions: For each question, please check the one answer which 

best describes your feelings: 

1. How much do you like foreign language classes compared to your 
other classes? 

I like foreign language classes better than most of 

my other classes 

I like foreign language classes about as much as most 

of my other classes 

I like foreign language classes less than most of my 

other classes 

2. If I had a friend who was a native speaker of the foreign 
language but who knew how to speak English. 

I would usually try to speak with him in the foreign 

language 

I would occasionally try to speak with him in the 
foreign language 

1 would really prefer to speak with him in English 



Although it is anticipated that some variation in questionnaire 
format and item phraseology will be required for questionnaires at the 
three achievement (and age) levels, the essential content of the 
questionnaires would not vary among the different levels. 



)ata to be collected from teachers . 

A questionnaire should be prepared that could be competed by a 
.ample of teachers at each of the test levels. The questionnaire should 
•onsist as far as possible of objective questions in which the respondents 
iould check answers corresponding to their personal opinions or 
observations, or make ratings along scales showing various degrees of 
opinion or preference concerning specified issue. 
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For the purpose of the present document, we think it will suffice 
to mention some of the more important questions or issues on which 
information should be obtained: 

Objectives of languaee teaching . What relative importance does 
the teacher attach to each of the four skills (understanding, speaking, 
reading, writing)? What relative importance is attached to purely 
utilitarian aims, such as conversing with speakers of the foreign 
language, listening to foreign broadcasts, reading books, corresponding 
with foreign peoples, as contrasted to auch objectives as attaining 
better appreciation of foreign cultures, enjoyment of literature in 
the foreign language, etc.? To what extent are the objectives of 
language teaching influenced by the foreign language requirements establis- 
hed by higher schools and college? By external tests and examinations? 

By pressure from school authorities, parents, etc.? Teachers should be 
encouraged to give their own opinions on these matters even when they 
may diverge from "official" points of view as promulgated by school 
authorities or professional associations. 

Methods of language teaching . To what extent, and at what stages, 
is each of the following aspects of language teaching methodology 
employed? 

Imitation and repetition of words and phrases 
Conversational exchanges between teacher and pupil, simple or 
complex depending on the level of instruction 
Use of objects, pictures, actions, gestures to convey msaning 
Memorization of dialogues 

Practice to perfect pronunciation of FL sounds 
"Pattern practice" to perfect mastery of grammatical structures 
Use of audio-visual aids, including films, tapes, language 
laboratory, etc. 

Study of grammatical rules and practice by written exercises 
Prepared or sight translations of sentences or paragraphs in 
French into the native language, either orally or in 
written exercises 
Frequent reviews and tests 

Again, teachers will be encouraged to give their opinions as to 
whether the methods they are using are those they would employ if given 
a completely free hand in the matter. 

Conditions of language teaching: 

Number and length (minutes) of classroom periods per week; amount 

of use of the language laboratory; extent of homework assignments 
Size of classes; opinions on optimal size of classes 
Selection of student for language study. Are all students required 
to study a foreign language? Is French one of several possible 
options? 

How are students encouraged and guided in their FL study? 

Are student grouped for ability? If so, how? 

How are texts and other materials selected: by the teacher? by 
local school authorities? other? 
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What texts are used? Are they adequate and appropriate? 

To what extent does the teacher prepare his own teaching materials* 

How many classes, and at what levels, are taught by the teacher? 
Are they all French classes, or are some of them in other 
languages or in subjects other than foreign languages? 

How proficient does the teacher regard himself to be according 

to MLA standards of proficiency level? What level of teacher 
training has he attained? Attended an NDEA Institute in 
French? Had foreign travel or study? 

Teacher 1 s evaluation of tests given in t his project: 

In terms of the objectives and content of the course, are the 
tests fair? 

Are they at an appropriate level of difficulty? Do they present 

techniques of testing to which the students are unaccustomed? 

Do the tests fail to measure important objectives? 

What proportion of their students would they estimate would be 

able to attain, say, 90 percent of the possible points on the 
tests being administered? 



SAMPLE TEST ITEMS 
Items for Listening Tests 

Listening tests can be prepared at almost any level of difficulty 
without use of the mother tongue, except for general directions pre- 
ceding the test. If it is desired to avoid the use of the written 
language, pictures can be used. Here is an example of an easy pictorial 

item: 



Instructions: You will hear a statement in French. The statement 

refers to one of the four pictures in your test 
book. You are to decide which one of the pictures 
is referred to. 

spoken on tape:) II y a des chaises dans le salon. 




(From MLA-Cooperative French Tests, Form LA, Listening, Item No. a, 
page 4.) 
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If there is no reason for avoiding the use of written language, other 
types of items are possible. The following are two easy items: 

(Instructions both printed in test booklet and spoken on tape) : 



You will hear several questions and statements dealing with 
everyday situations. For each one you are to select the best 
reply from among the four choices printed in your test book. 



(Spoken on tape) : 

+ 



(Printed in booklet) 
11 



(Spoken on tape) : j. 



(Printed in booklet) : 
12 



G'est anres la classe . (Pause) 

Numero onze. 

Qui est debout a c6te du bureau? 

Onze. (10 seconds) 

C'est anres la classe 
A Sur une chaise 
6 Des Livres. 

C Un drapeau 
D Le professeur. 

Numero onze. 

Que faites-vous quand vous quittez 1 ecole? 
Douze. (10 seconds) 

F A trois heures et demie. 

G Avez mon ami 
H Rien d f important 
J Dans 1* autobus 



(From MLA* Cooperative Tests, Listening Form 
LA, Items 11-12, page 5) 



Further items of the same general type, somewhat more difficult: 



(Spoken script) : 

Et maintenant. nos annonces publicitaires . (Pause) 

+ Numero trente et un. 

Voulez-vous un ben repas? Vous serez bien content chez Dupont. 
Tout y est d^licieuz: potages, viande, legumes et dessert. Le 
service y est excellent. 

Trente et un. (15 seconds) 

(Printed in booklet) : 

Et maintenant, nos annonces puhlicitaire3. 

A On parle d'un restaurant. 

B On parle d'une patisserie. 

C On parle d'un petit magasin. 

D On parle d'un voyage en avion. 



Fairly easy : 

Spoken on tape: 

+ 



Printed in booklet: 



Nous sonnies en voyage . (Pause) 
Numero sept. 

Vos billets, sil vous plait. 
Sept. (10 seconds) 

Nous sommes en voyage. 



7 A De nationality americaine 
B Les voici, monsieur. 
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Spoken on tape: 



C Non, pas de valises. 

D Mere! bien, monsieur. 

Numiro huit 

Mange- t-on vraiment bien au wagon-restaurant? 
Huit. (10 seconds) 



Printed in booklet: 

8 



F 

G 

H 

J 



Non, un wagon-lit. 

Des places vide®. 

En traversant deux wagons. 
Le mieux du monde. 



From MLA, Cooperative Tests, Form MA, Listening, 
Items 7-8, page, 4. 



Fairly difficult 



Spoken on tape: Directions: Maintenant nous ecoutons deux personnes qui 

se par lent. Dans cheque cas, chois issez la 
phrase qui pourrait suivre les remarques de 
la seconde per sonne. 



(Man A) "J &tais justemenet sur le point de^tourner a 
gauche quand ce camion nous a doubles." 

(Speed: Moderate -fast. Tone: Shocked.) 

(Man B) "Nous l'avons echappe belief Tu devrais faire 
attention aux autres voitures, mon vieux." 

(Spped : Moderate-fast. Tone: Shocked, then admonishing.) 

Printed in booklet: / 

(a) Je suis moins age que toi. 

(b) L' accident n'est pourtant pas grave. 

(c) Un moment de distraction, e'est tout. 

(d) Le camion etait arrete au bord de la route. 



From Description of the MLA Proficiency Tests for 
Teachers and Advanced Students, p. 14. 



A difficult "listening" item: 

The following dialogue is heard on the tape: 

Et- naintenant^ une petite scene dramatique. (Pause) 



Man 



Non, mon fils J& ?ues n'a pas assiste au bal hier soir, 
songe qu' a travaxller et a devenir officier. 



II ne 



Woman 

Man 



Evidemment e'est un garcon tres bien. 



Et sa soeur Annie est aussi tres bien, mais enfin, e'est une fille. 
On ne peut pas autant exiger d'une fille que d'un garcon. Les 
filles sont faites pour se marier, un point e'est tout. Je n'ai pas 
voulu qu'elle continue ses Etudes. Elle aurait voulu faire son 
droit. Le droit. Je lui ai dit: marie-toi. et apres, si ton mari 
autorise a faire ton droit, eh bien, ca lui regarde. Et Albert, 
votre garcon, qu 1 est-ce qu'il veut faire, lui? 
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Woman 



La medecine. . ,Hm! enfin il ne sait pas tres bien, au fend. 

A cet age on est hesitant. La literature 1' attire aussi. II 

♦* P a m ^ aef ft a c «n a1 4 



ecrit. Ce n'est pas mal, i... 



Man 



Ecrire! La jeuness d' aujourd'hui ne pense qu'a ca. De mon 
temp ... enfin, tout ca se resoudra. 



Woman 



Oh! surement. Avec la temps, n"est-ce pas..? Et cornme le dit 
mon mari: d'abord passer son baccalaureat, n'est-ce pas? 



Man 



D'abord. Et ne jamais les quitter de l'oeil. Moi, e'est ma 
methode. Avec Jacques elle m' a reussi. Respect. Discipline. 
Culture physique. II sera officier, Je lui fais apprendre le 
judo. II me remerciera un jour. Est-ce e'est votre methode aussi? 



Woman 



Euh. . . Oui. . . en somme. . . jusqu'a un certain point. Pour le judo 
en tout cas. 



Man 



C'est la bonne methode. Mais on ne voit plus nulle part de dis- 
cipline, de respect, de culture physique. La France meurt, 
madams, elle meurt, la France. 



Woman 

Man 



Peut-etre etes-vous un peu pessimiste. • • 



Pessimists? II y a cinquante ans qu'elle meurt. Mais naturellement 
per sonne ne s'en est encoure aper 9 u. 



Thus is immediately followed by spoken questions: 
Voici maintenant les questions (Pause) 



Num£ro trente-huit. 

Selon cet homme, h quoi les jeunes filles devraient-elles se consacrer? 
Trente-huit. (lo seconds) 



Printed in booklet: Et maintenant, une petite scene dramatique, 



38 



Spoken on tape: 



F A l'enseignement des jeunes. 

G Aux bonnes oeuvres. 

H A 1' emancipation de la femme. 
I Au mariage. 



Numero trente-neuf. 

Quelle est 1' attitude de la femme envers son 
fils? 

Trente-neuf. (12 seconds) 



Printed in booklet: 

39 



A Elle est plutSt tolerante. 

B Elle exige trop de lui. 

C Elle lui reproche d'etre pares seux. 

D Elle est fiere de ses progres & l'universite. 
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(Items 38-39 a MLA Cooperative Tests, Page 7, Listening, 
Form MA) 



SPEAKING ABILITY TESTS. 

Easy Level 

(Directions both printed and spoken:) 

You will ans wer a question about each of several pictures. Answer 

the question when you hear the signal. 

(Spoken on tape:) , » 

Que fait la jeune fille? (1 second, TONE, 8 seconds) 



The response is scored on a 3-point scale: 

3: Natural and meaningful response, free from error in vocabulary, 

grammar, and pronunciation (Response may or may not be a 
complete sentence). 

2: Meaningful response with minor error (s) in pronunciation. 

1: Response with major error (s). 

0: Irrelevant or unintelligible response j not attempted. 



The student is asked to imitate the following sentences spoken 
on the tape. In scoring, the accuracy of pronunciation of the underlined 
segments is rated: 



(Printed in booklet): 




(The examinee records his answer on tape in the 
8 seconds allowed.) 



Mimicry : 



Qu'il fait beau (o) Also scored for intonation. 
Allons chez eux (/*) 

Ils vont & l*£cole (e) 
etc. 







Reading Aloud: 

The student is asked to read the following passage aloud. In 
scoring the accuracy of pronunciation of the underlined segments is 
rated (the underlines do not appear on the text seen by the student) : 



>5 Hx | 



Deux amis, Paul et Jean, se rencontrent. 

Ecoute, qu'est-ce que tu fais de main ? x 

Demain je vais k onze heur es a la gare pour l'arrivee du train. 

Je vais rencontrer mon oncle. 

Tu as un oncle? 

Oui, c'est un demi-frere de maman. II est absent depuis six mois. 
et je ne le connais pas bien . 

Qu'est-ce qu.'il fait? 

II ecrit. 

(From MLA Cooperative Tests, French Form L, Speaking) 




Difficult speaking ability test: 




Instructions on tape: 

Here you see a simple story told in several scenes. When the 
signal sounds, tell the story that the pictures tell. Use as much 
imagination as you like. You may continue talking until you are told 
to stop. 

Pictures printed in booklet: 
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Easy: (Items from MLA Cooperative Tests, Form LA, Reading) 



Directions: Each of the foilwing sentences contains a blank space 
( ) indicating that a word or phrase has been omitted. From the four 
choices select the one which, when inserted in the ( ), best fits in with 
the meaning of the sentence as a whole. 



1 . 



"Maman, je meurs de faim! Donne-i 
done quelque chose a manger." 
"Mais voyons, mon enfant, est-ce 

A n'as pas ton cahier 
B veux un grand verre d.' eau 
C ne peux pas attendre le diner 
D preferes revenir domain 



23. Ma maison est solide. 
Elle est bdtie de ( ). 

tu ( )?" 

A glace 
B sable 
C feuilles 
D pierre 

23 ... 



2. Tu cherches toujours tes gants? 

As-tu pense A regarder dans ( )? 

F ton tapis 
G ton couvert 
H tes poches 
J tes bas 

2 ... 

3. Je vais faire venir le medecin parce 
que j'ai un ( ). 



24. Chut.' Pas de bruit! 
Par lex ( ) s'il vous 
pla£t. 

F a la fois 
G a haute voix 
H a tort 
J a voix basse 

24 ... 



A peu froid 
B gros rhume 
C appetit formidable 
D malentendu 



3 ... 
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Reading Test Items: Difficulty - Medium to Moderately Difficult 

(Items from MLA Cooperative Tests, Reading, Form LA, Items 26-31) 



Directions: Each of the selections in this part is followed by several 

questions or incomplete statements. First read the selection 
carefully. Then on the basis of the selection decide 
which choice best answers each question or completes each 
statement. 

"Philippe, va porter cette lettre au bureau de poste," dit la mere de 
Philippe. 

'Oh, mamanl Un seul jour de conge par semaine, et je dois faire 
des courses."' 

"Mais qu'as-tu, enfin? Tu seras de retour dans un quart d'heure." 

26. Philippe ne veut pas aller au bureau de post aujourd'hui parce qu'il 
F ne sait pas ou il se trouve 

G veut faire plaisir a sa mere 
H est facteur. 

J prefere s'amuser avee ses camarades 

26 ... 

27. La mere dit que Philippe pourra aller au bureau de poste et revenir en 
A un seul jour 

B t rente minutes 
C quinze minutes 
D cinq minutes 

27 ... 



28. 



Aux Etats-Unis, 
F lundi 
G mardi 
H vendredl 
J samedi 



cette conversation aurait lieu probablement un 



28 • • • 



Le matin, ma mere me reveille de tres bonne heure. Je m'habille rapidement, 
et je sors avec mon petit sac sur le dos, et le bout du nes et les oreilles 
bien caches dans une grosse echarpe de lalne, car on gele chez nous a cette 
‘epoque de l'annee. J' arrive presque toujours avant les autres. J'entre 
dans la salle encore vide, et je m' assieds a mon pupitre. 



29. Cette scene se passe au temps de 
l'annee ou il fait 
A doux 
B chaud 
C froid 
D bon 

29 ... 



30. Cet enfant se rend 

F au march£ 

G au byreau 
H a l'eglise 
J a l'£cole 

30 . 



31 






L' enfant arrive generalement en 
A dehors B arri^re C retard 



D avance 
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Reading Test Items: Fairly Difficult 

(Items from MLA Cooperative Tests 9 Reading, Form MA, Page 17, Items 33-35) 

Depuis leur mariage, voil& sept ans, Lucia et Miguel sont aux prises 
avec le meme probleme: r^gulierement , les journaux annoncent qu'il a 
decide de ne plus combattre, et qu'elle ne fera plus de cinema. Puis, 
aussi regulierement, on ecrit que Luis Miguel reprend la cape, pour une 
saison, et, aussitot, que Lucia fera peut-etre son nouveau film. Mais 
ce pauvre chantage de la femme an^ois&ee ne reussit pas. Apres quelques 
pourparlers avec un metteur en scene, Lucia comgrend vite que son mari 
descendra dans l'arene de toute facon, et que meme la perspective^de 
voir sa femme se replonger dans I'univers du cinema ne peut l'arreter. 

Quelle est la profession de Luis? 

A 11 est journaliste 
B II est 6toile de cinema. 

C II est toreador. 

D II est metteur en scene. 

Que pense Lucia du metier de Luis? 

F Elle en est enchant€e. 

G Elle le trouve inquiet ant. 

H Elle aime toute la publicite que lui donnent les journaux. 

J Elle le prefere au cinema. 

Que savons -nous de Lucia et de Luis? 

A Tous deux frequentent le cinema assidument. 

B Ce sont des chanteurs de grand talent. 

C Ce sont un mari et une fenme que se battent mutuellement. ^ 

D En depit de leur amour l'un pour 1* autre, Luis reprendra son metier. 



"Eh bien, ton ami Paul a de nouveau refuse de faire par tie de notre equipe 
de hockey." 

"Ne t'en fais pas! II finira par accepter. C'est qu'il aime a se ( ). 

F tirer d' affaire 
G montrer tout de suite obligeant 
H fier aux autres 
J faire prier un peu 



Reading Test Items: Difficult 

(From MLA Cooperative Tests, Form MA, Reading, Page 19, Items 48-50) 

II baissa la glace de la portiere. La fralcheur nue de cette ^poque de l'annee, 

l'odeur de la terre humide et celle, rassurante, des betes de fer me, une 

cloche grave entendue a travers des bois dont les feuilles jaunissaient. • • • 

C'€tait comme une eau froide et noire qui montait: Marc en fut 

submerge. Le soir tomb ait sans reverberes, sans fen£tres allumees: un 

soir sans homines, mais peupl€, bruissant, souverain, ou March ne se 

sen t ai t qu'un 'et ranger fragile. 



De quelle saison s'agit-il? 



F Le pr in temps 
G L'hiver 
1 H L'automne 

J L'et£ 

Ou se passe cette scene? 

A Pr£s d'un village 
B Sur un terrain de jeux 
C Dans une rue de petite ville 
D Au bord de la mer 

L'endroit lui 

F rappel le son enfance 
G parait bien agreable 
H inspire confiance 
J est peu sympathique 



Tests of Writing Ability 
Easy: 

Directions: Rewfite each of the following sentences so that the word or 
words printed on the line below it fit correctly into the new sentence. 
Make all changes necessary to produce a correct French sentence, but do 
not add any words and do not leave any out. Do not change the tense of 
the verbs. 

39. Cette nouvelle voiture est rapide et assez danger euse. 

. avion est 



40. Tu crois que tu peux y aller tout seul? Tu ne connais pas le cheminj 

V° us y aller tout seul? 

le cheminl 



(From MLA Cooperative Tests, French, Form LA, Writing, Page 22, Items 
39, 40) 
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A more difficult, "directed writing" test; it will be noted that the 
directions for this test require the use of the native language. 

Directions (58): On the separate answer sheet, write in French in 

connected* paragraph form an account for your friend of what you have 
been doing P since he left your house this morning and what your plans are 

for the rest of the day • 

Tell your friend: 

a at what time you got up in the morning 
b what you had for breakfast 
c that you stayed home until noon 

d that you spent the whole morning studying mathematics 
e that you went to school by bus at 12:15 

€ that vou found the examination easy but too long _ . , 

g that after returning home you will play tennis with some nen s 
h that after dinner you intend to listen to French records 
i that you have to review everything that is difficult for the French 

examination . 

j that by so doing you hope to pass with good nuuks 



(From the University of the State of New York, Regents High School 
Examination, French III) 
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Writing Test: Fairly difficult 



Directions: Write a paragraph telling the story suggested by the four 

pictures below: 







1 
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APPENDIX B-l 



IEA/B/55 
June 30th, 1968 



TEA NEWSLETTER 



Dear Colleague, 

1. Pre-testing 

(a) Cognitive measures 

During the past few months you have been inundated with a 
great amount of pre-testing requests. Unfortunately, some of the 
tests you received only in May. Thus, in those countries where the 
school year ends in April or May, pre-testing will have to be de- 
layed until August or September. Appendix I of this Newsletter 
gives the dates by which item analyses have been or will be returned 
by various countries to Hamburg. The last row on the chart gives 
the very final dates by which item analyses in the different sub- 
ject areas should now be returned to Hamburg. The International 
Committees will then meet in order to produce the penultimate forms 
of the tests. These will then be sent out for comment to National 
Centers and modified in the light of comments received. The final 
tests therefore should be ready some three months after the date 
of the return of item analyses as stated at the bottom of the chart. 

Thus it may be February or March before some of the tests are ready. 

This will mean that we shall have to change the original schedule 

for the dry run. 

(b) Affective and descriptive measures^ 

The form of pre-testing the affective and descriptive mea- 
sures has been slightly different from that of the cognitive measures, 
and is being carried out in two or more rounds. The following is a 
progress report on the various measures* 

(i) Science 

You have already received the report of the preliminary trial 
of Science attitude scales (IEA/ ATT/SCI/3) . The testing is now in 
its second round, and the scales "The Place of Science in the School 
and in the World" and "Interest in Science" are in their second round of 
testing in all countries at present. The descriptive scale Science 
in our School" is now in its first round of testing in all countries. 

There are two scales, (a) transfer and (b) interest, and the 
first round results have been analyzed and the second round will go 
out to all countries within two or three weeks. 

(iii) French , _ , 

The first round concerning ethnocentrism, attitudes the Frenc 

people and preference for won country have been analyzed and reported 
in IEA/ATT/FR/5. The second round is now out in all countries and 
concerns attitudes towards the French culture. 

(iv) E nglish 

The results of the first round of piloting proved to be un- 
satisfactory. The problem is being rethought, and we are hoping to 
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produce a general "Interest in English” scale. This will be circulated 
to all countries participating in English during the next fe a weeks. 

(v) Civic Education 

The following instruments have been produced: C.Q.l, C.Q.2, 

C.Q.3, C.Q.4, D.V.l, D.V.2, C.V.l, C.V.2, C.V.3, C.V.4, C.V.5, and 
C.V.6. It was hoped that these instruments would be piloted in the 
United States, England and Germany. So far, some results only have 
been received from England and Germany. It is expected that from the 
results of the pre-piloting a new set of instruments can be produced 
for full pre-testing by all countries participating in Civic Education 
in September of this year. 

(vi) General 

So far, four attitude scales, (a) I like school, (b) self es- 
teem, (c) achievement motivation and (d) stereotyped thinking, and one 
descriptive scale - school environment (containing two dimensions, 
authoritarian/permissive and subject /pupil orientation) have under- 
gone a first pilot round in England. Revised forms are now' being pro- 
duced and will be circulated to all countries for pre-testing within 
the next few weeks . 

It is hoped that the final pre-testing in all countries of all 
attitudinal and descriptive measures will take place in September, and 
that final instruments will be ready by December. 

2 . Delay in pre-testing 

Because we were late in getting tests out to National Centers, 
there has been a delay in getting results back to Hamburg. According 
to the attached list (Appendix I), it is clear that only Science, 
Reading Comprehension and Literature will be able to follow a fairly 
normal procedure and be ready by December. It is equally clear that 
delays in the pre-testing of French, English and particularly, Civic 
Education are occurring , which means that the original dates for the 
International Committee meetings cannot be adhered to. Most of the 
pre-testing for French and English will have to take place in the 
period September to November, which means that the meetings will not 
be held until later in the year. 

The English Committee must still produce the pre-tests for 
speaking and writing, and it is hoped that these will be ready in time 
to be administered sometime towards the end of 1968. In the case of 
Civic Education, it is possible that even longer delays may occur al- 
though it is hoped that pre-testing of both cognitive and affective 
measures can be carried out before the end of 1968. Similarly, many 
of the attitude and descriptive scales (both for specific subjects and 
across the board), as well as some questionnaires, will have to have 
a final pre-testing in September/October, and, hopefully, they will 
then be ready by the end of the year. It is clear that these delays 
have implications for the future program. 
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New deadlines 



All of the affective and descriptive measures will continue to 
be pre-tested during the year. In recent weeks various instruments 
have been sent to you for pre-testing and in the covering letter you 
have been requested to indicate the date by which you can return the 
punched cards to Dr. Choppin in London. It is recognized that in 
many countries it is too late to test during June/ July because of 
examinations, end of term activities, etc. Thus the instruments will 
have to be pre-tested in August/September/October. I would appreciate 
it if Heads of National Centers would respond to Hamburg indicating 
when punched cards may be expected to arrive on Dr. Choppin' s desk. 



As far as the. cognitive tests are concerned, the situation is 
as follows: 

(i) Science 



The International Science Committee Meeting is taking place in 
Hamburg from July 1st to 10th, and it is hoped that penultimate drafts 
of the final tests will be produced. It may be necessary to hold a 
small sub-committee meeting later concerning the Supplementary Science 
test materials (again it is anticipated that certain National Centers 
will be pre-testing these Supplementary Science Tests in September. 
Could this please be confirmed to Hamburg.) Every effort should be 
made to have the item analyses for the Supplementary Science Tests 
back in Hamburg by September 30th. 

(ii) Word Knowledge, Reading Speed and Reading Comprehension 

According to the information sent in by National Centers, it is 

anticipated that the item analyses for all of these pre-tests will 
have arrived in Hamburg by July 15th, 1968, They will then be collated 
and sent to Professor Thorndike for August 1st. 

(iii) Literature 

For the multiple choice booklets (1 to 6) it is now anticipated 
that all item analyses will be returned to Hamburg by July 31st 1968. 

(iv) French as a foreign language 

It is now anticipated that all item analyses can be returend to 
Hamburg by October 31st, 1968. 

(v) English as a foreign language 

It is now anticipated that all item analyses can be returned to 
Hamburg by October 31st, 1968. 

(vi) Civic Education 



It is now anticipated that all item analyses can be returned to 
Hamburg by October 31st, 1968, or possibly slightly later. 

(vii) Questionnaires 

It is anticipated that comments* and frequency distributions, 
etc. can be returned to Hamburg by October 1st, 1968. 

(vii i)National Case Study Questionnaire 

Unfortunately we are experiencing considerable delay in the 
production of the National Case Study questionnaire. It is, however, 
hoped that this can be sent to you within the next six weeks. All 
that will be required on this questionnaire will be comments. 



It would be appreciated if comments could be back by the end 
of August. 
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3. 



Amendments to IEA/B/53 



IEA/B/53 - a chart indicating which pre-tests are being ad 
ministered in which countries - was circulated on May 21st, 1968. 
Since then we have been notified of the following amendments: 

Australia - not testing Word Knowledge Population I 



Scotland - testing of SST 2 and 3 and French Population I doubtful 

Thailand - not testing Supplementary Science 

4. Future program for dry run and full testing 

Xt would appear that two alternatives are possible. 

(a) Despite the delays in pre-testing in English, French 
and Civic Education, an attempt is made to expedite 
the final stage of test construction and produce tests 
in these subjects in time for the dry run in 1969. 

In the case of the other subjects, there are no pro- 
blems in maintaining the original schedule. 

(b) To ensure that the best possible instruments are pre- 
pared, more time is devoted to finalising the test 
construction, that is, full time for comments and modi- 
fication, for French, English and Civic Education. 

This in turn implies that they would not be ready for 
the dry run in 1969, but could be available for such 

use in 1970. 

It is suggested that if alternative (b) is chosen, then the 
future program be amended to incorporate a further stage. Stage 2 
would consist of the dry run in 1969 for Science and Mother Tongue, 
with the full testing in 1970. Stage 3 would consist of the dry run 
and full testing for French, English and Civic Education at later 
dates to be agreed bu the Council. 

5 . Reasons for the dry run 

I have received several queries about the purposes of the 
dry run. ThAre are in fact two main purposes, which are: 

(i) to test procedures 

and (ii) to test computer programs 



Chile 

India 



not testing Reading Comprehension 
not testing Civic Education 



testing Reading Speed Populations I and II 
not testing Science Papulations IV and IVS 



Iran 



not testing Supplementary Science 
not testing Literature open ended 
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Re. (i) : It should be possible within two to three months of receiving 

the data for a population from a country to be able to raise queries 
with the country about possible errors in procedures (for example, in- 
correct coding, punching erros, etc.) 



Re. (ii) : The dry run will enable all programs to be tested with real 

data. This, it is hoped, will enable as many snags as possible to be 
removed . 



6 . 



Literature cross-over study 



In the spring of this year a study was made in England and the 
United States on multiple choice versus open ended items. Multiple 
choice items proved to be as reliable as open ended items and also ap- 
peared to be measuring the same thing. Although the reliabilities 
for the prose texts were much higher then for the poetry texts, they 
are still not very high (that is, about .65 for the prose and .22 for 
the poetry). A full report (IEA/TR/6) will be circulated within the 
next few weeks . 



7. 



Finance 



I am pleased to be able to inform you that the Volkswagen Found- 
ation has granted us $200,000 for Stage 2 of our work. 



8 . 



Proposed analyses for full testing 



Considerable work has been undertaken in the last few months 
in the planning of the proposed analyses. Two documents are at present 
being produced, one in lay terms and the other in technical terms, 
describing in some detail the proposed analyses. These will be circu- 
lated to all IEA members in September. 



9. 



Proposed administrative arrangements for full testing 



On May 27th you were sent a copy of IEA/B/54 which gives an out- 
line for the proposed administrative arrangements. It is most impor- 
tant that we get the reactions from National Centers to this document 
as soon as possible. May I, in your own interest, urge you to send in 
your comments to Dr. Postlethwaite in Hamburg as soon as possible. 



10 . Development of IEA data processing plans 



Dr. Bruce Choppin has begun work in London on the development 
of certain computer programs. Initially these are designed to handle 
the present pre-testing work and the development of the attitudinal 
and descriptive measures, but it is expected that towards the end of 
this year this work will be expanded to include more general statisti- 
cal programs for use in the next stage of IEA. It is difficult at the 
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moment to offer these programs for general use by National Centers, 
cause there are no supporting documents. 



be- 



ll. Amendments to T.EA Statutes 

In order that IE A becomes exempt from corporation tax in Ger- 
many, it has been found necessary to amend the IEA Statutes. The 
amendments are as follows : 



(a) Article 2 is now divided into two paragraphs. 

Insert "(a)" before the first paragraph. 

Insert "(b)" The co-ordination center of the Association is 
situated in Hamburg (Federal Republic of Germany). The co- 
ordinator is responsible for the day-to-day management of the 

Association . " 



(The explanation for this is that in order for IEA to be exempted from 
corporation tax in Germany, it must be regarded as a socially useful 
organization ("gemeinnutzig") , and to be so recognized must either have 
its official seat or its management seat in Germany.) 

(b) Insert in the last paragraph of Article 4 after "... forfeit 
all rights in the Association.": 

"They receive merely the amount of their paid-in capital and ^ 
the fair market value of their contributions in kind, if any.' 

(c) Insert a new Article after Article 22: 

"22a. In the event of profits being made by the Association, 
they may be used only for the aims stated in Article 1 
of these Statutes. The members may take no share in 
the profits, and in their capacity as members may re- 
ceive no gift, donation, grant or loan from the Asso- 
ciation. The Association may not incur administration 
expenses foreign to the aims of the Association or pay 
disproportionate amounts (for example, salaries, allow- 
ances, honoraria, etc.) to any one person." 



Delete "its assets" at the end of the second paragraph of Article 
24 and insert the following: 



" . . . the assets remaining after the repayment of 

(a) the paid-in capital, if any, and (b) the fair 
market value of the members’ contributions in kind, if 
any. The remaining assets may only be used for aims in 
accordance with Article 1 of these Statutes. 



B-6 



| 



'I 

t 



i 

l 



I 










>* 







afejMCr'kArlMN 






The decision of the Council regarding the use of the 
remaining assets requires the approval of the compe- 
tent tax authority." 

A formal decision will be required at the December Council Meeting 
to amend the Statutes, (Will Council members please regard this as a 
formal notification (as required by Article 24 of the Statutes) of 
amendments to the Statutes. It is still hoped that we may get a spe- 
cial exemption from the German tax authorities without changes to the 
Statutes, but should this fail we must be in a position to change 
the Statutes in December.) 

12 . TEA Publications 

You will be pleased to hear that a short version of the IEA 
Mathematics volumes has appeared in Germany under the title: 
"Leistungsmessung in der Schule: Eine Internationale Untersuchung am 

Beispiel des Mathematikunterrichts". It is published by the Verlag 
Moritz Diesterweg, Frankfurt and costs DM 14, — . A short version in 
Swedish, written by Mats Hultin, will be coming out in the very near 
future . 



Yours Sincerely, 



Torsten Husen 
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IEA/B/74 

31st December 1968. 



TEA NEWSLETTER 



Dear Colleague, 

May I take this opportunity of wishing you all the best for 1969 - 
our dry-run year. A great deal has happened in the last three months and 
without going into great detail, I shall try to summarize the points of 
interest to all persons involved in IEA. 

1. New Centers 



At the IEA Council meeting in Hamburg in December 1968, two new re- 
search institutes were admitted to the IEA Council. These were: 

a) The National Institute of Pedagogy, Budapest, Hungary. 

b) The Curriculum Unit of the Department of Education, Wellington, 
New Zealand. 

This now makes 19 research institutes actually participating in IEA 
and the IEA Council decided that for the time being the number of research 
institutes should be restricted to twenty, but that if further applica- 
tions are received, the Council should review its policy of admission of 
further centers in terms of the general enlargement and costing of IEA’s 
work . 



2. Change of membership of Standing Committee and Bureau 

The terms of office of Professor Bloom (Chicago), Dr. Walker (Edin- 
burgh) and Professor De Landsheere (Liege) on the Standing Committee have 
come to an end, and Professor Bloom indicated his desire to resign from 
the Standing Committee and also from the Bureau. Dr. Walker indicated his 
dssirs to resign from the Standing Committee. The Council, therefore, 
elected Professor Foshay, (New York) and Dr. Ayman (Tehran) each for a two- 
year period, and re-elected Professor De Landsheere to the Standing Commit- 
tee for a further two-year period. The Standing Committee elected Profes- 
sor De Landsheere to the Bureau for a one-year term of office. These 
changes of membership formally take place on April 1st, 1969. 

3. New Chairman of the International Literature Committee 

Professor Foshay has indicated his desire to resign as Chairman of 
the International Literature Committee, although he will continue as a 
member of the Committee. The Bureau has invited Professor Alan Purves 
of the University of Illinois to be the new Chairman, and Professor Purves 
has accepted. His appointment takes effect from January 1st 1969. 

4. Revision of IEA Statutes 

Certain minor changes have been made to the IEA Statutes (see June 
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1968 Newsletter) by the Council so that IEA can be recognized as corpora- 
tion tax free in the Federal Republic of Germany. 

5. Stage 2 Tests 

The test instruments for Stage 2 have now been completed. They are 
as follows: 



No. 


Test No. of 

Items 


Time 

(Mins.) 


Optional 


Science 










IEA/ 1A 


Pop. I 


20 


30 




IEA/ IB 


Pop. I 


20 


30 




IEA/4A 


Pop. II 


40 


60 




IEA/4B 


Pop. II 


40 


60 




IEA/10A 


Pop. IV 


36 


60 




IEA/11B 










(P-U) 


Pop. IV 


30 


60 




IEA/15 


IVS Bio 


40 


60 


Optional 


IEA/16 


IVS Che 


40 


60 


Optional 


IEA/17 


IVS Phy 


40 


60 


Optional 


IEA/19 


II Practical 


9 


90 


Optional 


IEA/20 


IV Practical 


7 


90 


Optional 


IEA/21 


IV " (Bio) 


5 


90 


Optional 


IEA/ 2 2 


IV " (Che) 


3 


90 


Optional 


IEA/23 


IV ” (Phy) 


7 


90 


Optional 


Reading 


Comprehension 








IEA/3C 


Pop. I 


21 


25 




IEA/ 3D 


Pop. I 


24 


25 




IEA/3P 


Pop. I (Practice) 


40 


10 






Reading Speed 








IEA/ 3 J 


i? ii 


40 


10 




IEA/7C 


Pop. II 


26 


50 




IEA/7D 


Pop . II 


26 


50 




IEA/7P 


Pop . II (Practice) 


40 


10 






Reading Speed 








IEA/7J 


ii m 


40 


10 




XEA/14C 


Pop. IV 


25 


50 




IEA/14D 


Pop. IV 


29 


50 




Word Knowledge: 








IEA/2F 


Pop. I 


40 


10 




IEA/6F 


Pop. II 


40 


10 




IEA/13F 


Pop. IV 


40 


10 




Literature 








IEA/8X 




39 


50 




IEA/8Y 




38 


50 




IEA/8Z 




39 


50 




IEA/9W 




37 


50 




IEA/ 18 




1 


60 


Optional 



Bulletin of Erratum 
Sheet 



Pink Bulletin and erra- 
tum sheets 1 and 2 (pink) 



Blue Bulletin and erra- 
tum sheet no. 



Yellow Bulletin 

II II 



II M 



Blue Bulletin and erra- 
tum sheets no. 1 and 2 
(blue) 
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No, 



Test 



No. of Time Optional Bulletin or Erratum 
Items (Mins . ) Sheet 



Ques tionnair es 

Student General 
IEA/2G Pop. I 

IEA/6G Pop. II 

IEA/13G Pop. IV 



40 

41-79 

41-79 



See Manual 1 
for times of 
these 



Yellow Bulletin and er- 
ratum sheets no. 1, 2, 
3, and 4 (yellow) 



IEA/5S 


Pop. II 


28 


IEA/12S 


Pop, IV 


28 


Student Literature 




IEA/9Q 




35 


IEA/TQ1 


Teacher General 


38 


IEA/TQ2 


" Science 


21 


IEA/TQ3 


n Reading 






Comprehen. 


23 


IEA/TQ4 


" Literature 


38 


IEA/SQ1 


School Question- 






naire 


33 


Attitude 


and Descriptive Scales 


General: 






IEA/2H 


Pop. I 


36 


IEA/6H 


Pop. II 


36 


IEA/13H 


Pop . IV 


36 


Science: 






IEA/5T 


TOUS Pop. II 


15 


IEA/12T 


TOUS Pop. IV 


15 


IEA/1K 


Pop. I 


22 


IEA/5K 


Pop . II 


48 


IEA/12K 


Pop . IV 


48 


Literature: 




IEA/9R 




20 



See Manual 1 



Yellow Bulletin and er- 
ratum sheets no. 1, 2, 
3 and 4 (yellow) 



Erratum Sheet No. 2 (blue) 
The optional tests will be given by the following National Centers: 



Science IVS: England, Fed. Rep. of Germany, Hungary, Iran, Japan, New Zealand, 

Scotland, Sweden. 

Science Practical: „ >T * i a 

Pops. II and IV: Chile, England, Iran, Italy, Japan, New Zealand, 

Scotland. 

Pop. IVS: Chile, England, Iran, Japan, New Zealand, Scotland. 

Literature IVS: National Centers have not yet indicated whether or not they 

will be participating in this. Could Heads of Centers please 
write to Dr. Postlethwaite as soon as possible? 



State of Attitude and Descriptive Scales 



As a result of a long discussion at the Council meeting, it was 
decided that the work on the F scale and the self-esteem scale should be 
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droppe d . Some changes to the Science scales (in the yellow bulletin) 
have been made and appear in Erratum Sheet No. 1 (yellow). * 



out: 



At the beginning of January the following scales have been sent 



IEA/2H, 6H and 13H 

General for Pops. I, II and IV 



No. of I terns 



Erratum Sheet 



IEA/9R Literature 



36 each 
20 



No, 3 (yellow) 
No. 2 (blue) 



end • j* r \? h ° PP 5 n haS also P re P ared accompanying notes for each set of 
lnd J Catins * he i nt ent of each item. National Centers should take 
to h m ° thSSe n ° teS When translating the items. (This also applies 

the^English C ° UntrleS " here m ° ther t0ngUe * S SOme f °™ ° f 
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6. Dry Run for Stage 2 j 

U 

The instructions for the administration within countries are j 

included in Manuals 1, 2 and 3 in the Green Bulletin produced in December j 

1968. • : 



The following table indicates 
carrying out the Dry Run Testing: 


at which time National 


Centers 


are 


i 

l 

I 




Population I 


Population II 


Population III Pop'n 


IV & IVS 


i 

! 


National 

Center 


Dry 

Run 


Main 

Test-r- 

ing 


Dry 

Run 


Main 

Test- 

ing 


Dry 

Run 


Main 

Test- 

ing 


Dry 

Run 


Main 

Test- 

„Z2S 


•: 

i 






mm 


7/69 


7/70 


- 


- 


7/69 


7/70 


i 




3/69 


3/70 


3/69 


3/70 


- 


- 


3/69 


3/70 


j 




10/69 


10/70 


10/69 


10/70 


- 


- 


10/69 


10/70 


i 

■i 




6/69 


6/70 


3/69 


3/70 


3/69 


3/70 


3/69 


3/70 


l 


Fed. Rep. of 


5-6/ 

69 


5-6/ 

70 


5-6/ 

69 


5-6/ 

70 


- 


- 


5-6/ 

69 


5-6/ 

70 


? 

i 


Finland 


3/69 


3-4/ 

70 


3/69 


3-4/ 

70 


3/69 


3-4/ 

70 


11/69 


1-2/ 

70 


l 


France 







— 


— 


- 


- 


4/69 


4/70 


1 




5/69 


5/70 


5/69 


5/70 


5/69 


5/70 


5/69 


5/70 




India 


9/69 


2/70 


9/69 


2/70 


- 


- 


5/69 


2/70 






2/69 


2/70 


4/69 


4/70 


4/69 


4/70 


2/69 


2/70 




Italy 


2/69 


3-4/ 

70 


2/69 


3-4/ 

70 


- 


- 


2/69 


3-4/ 

70 


j 

$ 






4/70 


4/69 


4/70 


— 


- 


10/69 


10/70 


$ 


Netherlands 


Dry run: Planned for 


March, 

1970 


if not possible Oct. 1969 


j 

$ 

p 








7/69 


7/70 




- 


7/69 


7/70 








5/70 


5/69 


5/70 


- 


- 


__JZ§L 


3/70 




Scotland 


6/69 


5-6/ 

70 


6/69 


5-6/ 

70 


- 


- 


6/69 


3/70 


1 




5/69 


5/70 


5/69 


5/70 


5/69 


5/70 


5/69 


5/70 


| 

a 

| 




6/69 


12/69 


6/69 


12/69 


7/69 


12/69 


7/69 


12/69 


i 






4/70 


4/69 


4/70 






4/69 


4/70 


i 

■'j 


The next table gives 
to Stage 2 testing: 


the final commitments made by National 


Centers 


• 

i 
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Science Reading Literature 

_____ Comprehension 



\ 




I 


II 


III 


IV 


IVS 


I 


II 


III 


IV 


II 


III 


IV 


IVS 


t 


Australia 


X 


X 




X 
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7. Stage 3 Tests 

Work is continuing with the preparation of instruments for French as a 
foreign language, English as a foreign language and Civic Education. Penulti- 
mate versions of reading comprehension and listening comprehension French tests 
for Populations I, II, IV and IVS have now been prepared and these will be re- 
pretested between February and May 1969 in England, Iran, Italy, the Netherlands 
Sweden and the U.S.A. New writing and speaking tests will be prepared and will 
be prepared and will be pre-tested at the same time in the same countries, as 
well as in Chile. The final versions of all French instruments will be prepared 
in early July at the next meeting of the International French Committee. 

The item analyses for Civic Education are nearly all in, and proposed 
final tests will be prepared at the end of February by a sub-committee of the 
International Civic Education Committee. More work will be required in pre- 
testing the attitude scales in Civic Education throughout 1969. In the mean- 
time, the Civic Education Committee has produced a revised rationale for its 
testing in "The Assessment of Civic Education" (IEA/CIV/12) . 
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The item analyses for English as a foreign language are coming in 
slowly. However, the writing and speaking tests have not yet gone out to 
National Centers, but it is hoped that they will go out in the course of the 
next three weeks. Fortunately, since the dry run testing for this subject is 
not to take place until 1970, we have slightly more time to complete the pre- 
testing. 



8 . 



Mathematics Data Bank 



The data in the mathematics data bank can now be processed on the 
IBM 360 systems as well as on the 7094 Computer at Chicago. The data will 
be made available by IEA both at the Stockholm computer and the Columbia 
University computer. All requests should be sent to the Chairman of IEA: 



Professor T. Husen, 

I .E.A. , 

c/o Unesco Institute for Education, 
2000 Hamburg 13 
Feldbrunnens trass e 70. 

Federal Republic of Germany. 



All approved requests will be carried out by the IEA Data Processing 
personnel at cost. 



9. 



IEA Item Bank 



IEA has, in general, pretested about 3-4 times as many items as go 
into its final tests with the result that many ’good’ items are not used. 
It is considering the possibility of establishing an item bank where these 
would be available for use by bona fide researchers. 



10. 



IEA Personnel and Locations 



John Hall of the N.F.E.R. joined the IEA staff on January 1, 1969, 
and will be working with Bruce Choppin on the data processing. They will 
continue the development work until the end of March 1969, in London and 
move to New York at the beginning of April. Mr. Richard Russell of London 
will be working for the first six months of 1969 as a programmer. 



Neville Postlethwaite has been given the title of "Executive Director" 



It is still unclear whether the IEA Headquarters will remain in Ham- 
burg. The Unesco Institute will have the whole matter of IEA's accommoda- 
tion discussed at its Governing Board meeting in April, whereafter a decision 
will be made. 



11. IEA Visiting Scholar Center 



The Swedish Government has very kindly given IEA a suite of offices 
in the Wenner Gren Center in Stockholm. These offices will be used partly 
by the Swedish National Center and partly by IEA International. At the same 
time educational research scholars working on an IEA problem will be invited 
to Stockholm for short periods of time (1-6 months) as visiting scholars. 
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and will also be provided with personnel accomodation in the Wenner Gren 
apartment complex. 



As usual, if you have any queries on the content of this newsletter, 
or if you should require further information, I would appreciate it if you 
would contact either Dr. Postlethwaite or me. 



Yours sincerely, 



Torsten Husen 
(Chairman of IEA) 
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APPENDIX C 




INTERNATIONAL ASSOCIATION FOR THE EVALUATION 
OF EDUCATIONAL ACHIEVEMENT (IE A) 

STATUTES 
Name and aims 

1. An international, non-profit-making, scientific association is 
constituted under the name of the "International Association for 
the Evaluation of Educational Achievement" (Hereafter known as IEA) , 
the principal aims of which are:. 

(a) to undertake educational research on an international scale; 

(b) to promote research aimed at examining educational problems 
common to many countries, in order to provide facts which can 
help in the ultimate improvement of educational systems; 

(c) to provide, within the framework of the Association, the 
means whereby research centers, which are members of the 
Association, can undertake cooperative projects. 

Seat and legal status 

2 (a) The seat of the Association is in Liege (Belgium). The Associa- 

tion is constituted in accordance with the Belgian law of the 
25th October, 1919, regarding international non-pi;ofit-making 
scientific societies, modified by the law of the 6th December, 

1954. 

(b) The co-ordination center of the Association is situated in Ham- 
burg (Federal Republic of Germany). The co-ordinator is res- 
ponsible for the day-to-day management of the Association. 

Composition 

3. Membership of IEA is restricted to institutions carrying out research 
in education (participating centers). Any such research center which 
has the status of a body corporate is eligible for membership, 
provided that it is sufficiently qualified and equipped with the 
resources required to undertake the research projects envisaged. 
Individuals working on the project may also be admitted as members. 
Membership is decided by the General Assembly (defined below). This 
decision requires a three quarters majority vote of the members of 
the General Assembly present. 

4. Membership can be terminated: 

(a) by a three quarters majority vote of the members of the General 
Assembly present, where there are serious grounds; 

(b) by a vote of the same majority if a member ceases to satisfy 
the conditions of participation. 
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Those ceasing to be members forfeit all rights in the Association. 
They receive merely the amount of their paid-in capital and the 
fair market value of their contributions in kind, if any. 

Organs and Administration 

5. The organs of IEA are the General Assembly, the Standing Committee 
and the Bureau. The Standing Committee has its seat in Liege, 

Belgium. 

6. The General Assembly, which comprises a representative from each 
participating center as well as individuals who may be admitted under 
paragraph 3, has complete authority. Each representative has one 
vote only. The General Assembly meets, in principal, once a year 

in September, and determines the general policy of the Association. 

It examines the reports and reviews the expenditure of the previous 
period and adopts the program and the budget for the following 
period. Each period lasts from 1st September to 31st August. 

7. The General Assembly can establish such subsidiary organs as are 
necessary, either of a temporary or permanent nature. It determines, 
by a three quarters majority vote of all its members, the general 
composition, authority and term of office of these organs. 

8. The General Assembly elects, by a three-quarters majority vote of . 

its members present or represented, a Chairman. 

9. The General Assembly elects, by a three-quarters majority vote of 
its members present or represented, a Standing Committee. 

10. The Standing Committee comprises the Chairman and six other members, 
all elected by the General Assembly, which also has the power to 
dismiss them. 

The duration of the term of office of the Standing Committe members 
is normally three years. There should always be at least one 
Belgian member on the Standing Committee. 

11. At the first election of the Standing Committee, one half of the 
members of that committee will be elected for a period of two years 
and the other half for a period of three years. Subsequent elections 
will be made for a period of three years. 

A member of the Standing Committee may be dismissed by the General 
Assembly, on the existence of sufficient grounds, by a three-quarters 
majority vote of its members. 

12. The Standing Committee will be convened annually, or more frequently, 
at the discretion of the Chairman. 

13. The functions of the Standing Committee are to make decisions, between 
meetings of the General Assembly, within the framework of the overall 
policy determined by the General Assembly; to appoint the Technical 
Director, the members of the sub -commit tees, staff and consultants, 
and to make decisions concerning expenditure within the limits of the 
budget determined by the General Assembly. 
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14. The Bureau comprises the Chairman and two members of the Standing 
Committee elected by that Committee. 



15. The Bureau is responsible for the execution' of decisions made by 
the General Assembly and the Standing Committee. 



Authority 



16. The General Assembly has the authority to make valid decisions when 
half of its members are present or represented. If the quorum is 
not reached, a second General Assembly, convened 12 weeks later, 
with the same agenda, may take decisions on the matters in hand with 
a three quarters majority of those present. 



Each member may register a proxy vote for another member, but only 
one other. 



All decisions made by the General Assembly, other than those stipu- 
lated in paragraphs 3, 4 and 25, require a simple majority vote of 
the members present. 



17. The Standing Committee has the authority to make decisions when at 
least three of its members are present. 



All decisions made by the Standing Committee require a dimple majority 
vote of the members present. 



18. All documents involving the Association are, with the exception of 
special procurations, signed by the Chairman, who will not be 
required to justify his powers to any third party. 



19. In all judicial actions undertaken by the Association, either as 
plaintiff or defendant, and in all prosecutions and lawsuits, 
the Association will be represented by the Chairman or one of the 
members of the Standing Committee appointed by the Chairman. 



20. The resources of IEA comprise funds arising, for example, from 
gifts, grants, legacies, and profits from the sale of its 
publications. The General Assembly may also determine the amount 
of an annual contribution. 



21. The IEA national costs will normally be secured by the respective 
research centers themselves. 



22. IEA will finance its international costs by means of the resources 
listed in paragraph 20. 



22a. 



In the event of profits being made by the Association, they may 
be used only for the aims stated in Article 1 of these Statutes. 
The members may take no share in the profits, and in their 
capacity as members may receive no gift, donation, grant or loan 
from the Association. The Association may not incur adminis- 
tration expenses foreign to the aims of the Association or pay 
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disproportionate amounts (for example, salaries, allowances, honor- 
aria, etc.) to any one person. 

The auditing of accounts will be carried out each year by a competent 
international authority and the accounts are to be approved by the 
General Assembly. 



Modifications and Dissolution 

24. All proposals for modification of the statutes and for dissolution 
of the Association must come from one or more members of the General 
Assembly. All members of the General Assembly should be notified 

of such proposals (either by the Chairman, or in the absence of the 
Chairman by the other members of the Bureau) at least six months 
before the meeting of the General Assembly during which they are 
to be discussed. No decision on these matters can be taken without 
a three quarters majority vote of the entire General Assembly. If 
the quorum is not attained, a second General Assembly convened 12 
weeks later with the same agenda, may take decisions on the 
matters in hand with a three quarters majority vote of the members 
present. 

In the case of dissolution, the General Assembly decides, by the 
same majority, on the steps to be taken for the liquidation of the 
Association and on the future of the assets remaining after the 
repayment of: 

(a) the paid-in capital, if any, and 

(b) the fair market value of the members' contributions in kind, 
if any. 

The remaining assets may only be used for aims in accordance with 
Article 1 of these Statutes. 

The decision of the Council regarding the use of the remaining 
assets requires the approval of the competent tax authority. 

25. Any item which has not been provided for by the present statutes 
will be dealt with according to the provisions of Belgian law on 
the matter in question. 
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FIRST GRID OF TOPIC AREAS AND OBJECTIVES IN SCIENCE 



Subject Areas 



Earth Sciences 



1. Solar System 

2. Stellar Systems 

3. Meterology 

4. Earth's Crust 



5. Physical Geography 
and Geology 



6. Soil Sciences 



Biological Sciences 

7. Cell Structure and 
Function 

8. Homeostasis 

9. Cell Metabolism 

10. Cell Responses 

11. Concept of the Gene 

12. Diversity of Life 

13. Metabolism in 
Organisms 



14. Regulation in 
Organisms 



Objectives 
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APPENDIX D-2 



TABLE 1 



Expanded List of Teac hing Objectives 



Objective aa it appears on 
the form 



Explanatory expa nsion of the 
objective 



1. 



Obtaining scientific 
information 



1 . 



The acquisition of scientific 
knowledge 

i) Observation of phenomena 
and its description in 
the pupils* own language* 

ii) Growth of vocabulary; 

scientific terminology as 
giving precision to 
scientific description* 



iii) Quantifications measure- 
ment as an extension of 
descriptions the selection 
and use of instruments; the 
appropriateness of measures s 
estimations and limits of 
accuracy* 



iv) Experimentation including 

the recognition of a problem, 
formulation of working 
hypotheses , selection of 
suitable methods and the 
devising of appropriate 
experimental procedures* 



2 . 



Interpreting scientific 
information 



2. 



The organisation, interpretation 
and use of scientific knowledge 



i) Organisations classifica- 
tion and processing of data 
and the recognition of re- 
lationships* 



ii) Presentation of data* 



iii) Generalisation, which in- 
cludes extension of termin- 
ology in the form of defini- 
tions s functional relation- 
ships and their expression 
as scientific laws* 
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3. Theorisation - 
construction 


3* The building up, formulation, use and 
refinement of theoretical models 
(mental pictures or concepts) 




i) The recognition by the pupil of 
the need for a theoretical model 
in order to relate different 
phenomena. 




ii) The recognition and appreciation 
of the relationships that are 
satisfied by the model. 




iii) The ability on the part of the 
pupil to make rational predic- 
tions following the use of the 
model. 


4, Theorisation - 
utilisation 


4. The use of theoretical considerations 
to direct observation and experiment. 
Evaluation and interpretation of ex- 
perimental results at the theoretical 
level. 


3* Comprehension 


5* The ability to identify, comprehend, 
and recognise the significance of 
scientific knowledge in a new con- 
text whether it be obtained as the 
result of direct experience or de- 
rived from secondary sources. 


6. Application of scien- 
tific knowledge 


6. The application of scientific knowl- 
edge in a new and wider context and 
across subject boundaries. 


7* Personal and social 
objectives 


7* The development of personal skills, of 
hand and mind, and the encouragement 
of desirable attitudes and habits. 
Science in its historical perspectives 
Technical and economic aspects of 
science: the relationships between 

scientific progress, technical achieve* 
rnent and economic development. Con- 
temporary social aspects of science at 
the personal, communal, national and 
international levels, and especially 
the idea of human responsibility. 


8 • Philosophical aspects 


8. The limitations of scientific explana- 
tion; Scientific methodology. Influ- 
ence of the philosophy of science upon 
general philosophy. 
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mouTTTT r.ATIOH OF GRID 0 1 TOPIP -S AND OBJECTIVES 
Letter from Iij^ernatlonal Cgm mlttee to Jjatlonal Committees, May 19 61 



Completed grids have been received from ten National Centres together, 
in most cases, with comments of great interest and valuer 



Bulletin No. 2 Phase II has also been issued; it redefines the target 
populations (pages 2 and 3) and sets out the testing Procedures (pages 1 * 
and 5). These sections should be studied before any action is taken as 
a result of this report and in accordance with the timetable given in 
IEA/SCI/1, April-September, 1967. 



Much of the success of the Project, so far as science is oonc^nuu. rests 
upon the production of suitable test items by all the contributi^ co^- 
tries. OMs must be our main and urgent task during the next three months 
in order Vo save time and effort, lists are attached of subject areas which 
are common to all, or nearly all, the participating countries for the 
groups who will be tested. These have been prepared after an analysis 
of tte grids by the Chairman of the International Jhis 

and without prior consultation ^th^ther members of the Coatee. This 

has been done in order to gain precious time and it is hoped that the 
concentration of effective effort that can result from this rather un- 
democratic action will justify it and offset any imperfections revealed 

in the lists. 



It can be said that the grids have served their purpose in spite of their 
obvious weakness about which the International Committee were onl y*°° 
well aware. The purpose was to discover common subject areas ° n w hic £ 
to base the testing instruments, and to gain some impression of the o - 
jectives science teachers sought to achieve so that attainment could be 
assessed against these objectives. 



It was apparent when the grids were analysed that the form in which they 
were given was much better suited to some countries than to others. Not 
unexpectedly the shortened list of teaching objectives came in for some 
criticism, as did the way in which the subject areas were expressed and 
arranged. Some contributors thought the list of objectives was too con- 
densed so that discrimination was lacking, while others thought it was 
too long because teachers were not usually aware of the objectives they 
aimed to reach in their teaching. 



The subject areas given in the attached lists should prove sufficient 
for the construction of the tests as there was a fair measure of agree- 
ment in the grids. The same cannot be said of the objectives, in which 
very little consistency could be detected even in apparently fairly ob- 
vious cases. It would seem wise therefore to adopt the following pro- 
cedure in the construction of test items: 
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1, To devise test items on the subject areas contained in the attached 
lists and designed to test, as far as possible, the achievement of 
specified objectives, 

2. To submit these test items to groups of experienced science teachers, 
asking them to specify what objectives in their opinion the items 
assess, 

3. To send the test items to IEA. Hamburg, indicating the target popula- 
tion for which they are intended and the objectives they are thought 
to assess. For the time being, it might be wise to use the objec- 
tives as amplified in Table 1 IEA/SCI/3 as a basis, but it is to be 
hoped that National Committees for Science will feel free to enlarge 
andc'i modify that list, which will be revised later in the light of 
the suggestions and comments which have been, and will be, received, 

4, In general, objective type tests which can be marked automatically 
are looked for and the International Committee will be particularly 
pleased to receive test items of a new kind designed to assess 
specific scientific abilities, but this should not inhibit the 
production of items of a subjective character so long as these are 
felt to be indispensable to the proper assessment of a given ability, 

3* The examination of the pre-tests by the National Committees during 
the period October/November 1967 should ensure that the testing in- 
struments cover an adequate sample of the subject areas, measured 
agginst the teaching objectives, for satisfactory results to be ob- 
tained, But, as was stated earlier, the success of the whole opera- 
tion depends to a very high degree upon the skill, energy and ingenu- 
ity displayed by the National Committees working in conjunction with 
a wide circle of practising science teachers. The active co-operation 
of National Committees is therefore earnestly sought. 
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List of Subject A reas^Suggested_for_the^Production_of_Test_Items 

Target Population (l) 10+ (see Bulletin 2 pages 2 and 3) 



Topic 



Ref. in 
Grid 



Comment 



Earth Science 



The Earth in 1 

the Solar System 
Weather Studies 3 

Physical 

Geography 5 

Biological Sciences 



Diversity of 

Life 12 

Breathing, feeding 
and the elimina- 
tion of waste 13 

Reproduction 16 

Natural Environ- 
ment 18 



The general impression gained from the grids 
and accompanying comments is that at this 
stage earth science is taught as part of the 
general exploration of the natural environ- 
ment , often as geography. Test items should 
be constructed with this informal approach 
in mind. The growth of a more precise vo- 
cabulary to describe familiar phenomena 
would be a legitimate objective to test. 

Most countries will not have introduced ex- 
perimental science in anything but a very 
restricted sense. 



Although human biology as such is not often 
taught, the growth of biological ideas is 
generally seen in relation to human life and 
man's activities. 
















ERIC 



Topic 


Hef. in 
Qrid 


Comment 


Chemistry 




Few countries give any substantial attention 


Chemical 

Change 


23 


to chemistry at this stage and yet important 
chemical ideas, e.g., of chemical substance, 
extraction, separation, purification, and 
chemical properties must be emerging, so 
that items might be set under these two topi< 
headings. 


Haw Materials 


30 




Physics 




Measurement, although perhaps a more obvious 


Measurement 


33 (see 


part in physics, is misplaced here and 
should be viewed in a more general way as 
applying to all the sciences. 


Motion 


comment) 

36 




Forces and their 


applications 


37/38 




Floating and 


Sinking 


4o 




Expansion and 


Contraction 


41/42 




Melting and 
Boiling 

Mirrors 
and Lenses 


44 




Sound especially 
Musical Instru- 
ments 


^3 




Elementary 


Electricity 


48/49/50 
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Target Pop ul ations (2 8c 5 ) (see Bulletin 2 pages 2 and 3 ) 




It appears that there is a fair amount of 
common ground at these levels but there are 
indications that the depth of treatment and 
the extent of the practical work vary con- 
siderably. It is suggested that test items 
should be based on a quantitative treatment 
using simple numerical examples when they 
serve to illustrate fundamental principles 
and to establish relationships more clearly. 
Qualitative relationships and test items 
are not, however, to be omitted entirely. 



Biological Sciences 



The cell as a unit 
of structure 
and function 
(simple 7 / 9 / 

outline only) 10 



Diversity of 
life and 
inter-rela- 
tionships 

Biology of 
selected 
organisms to 
illustrate 
characteristics 
of life 



12 

With some reference to human beings. 

13/14/ Relation of structure to function. Adaptations 
I5/I6 to mode of life. Elementary anatomy and 

physiology of representative plants and animals. 



National en- 
vironment and 
seasonal 18 

rhythms 



Chemistry 




Elementary chemistry can be tested at this 
stage, e.g., notions of classification ele- 
ments, compounds, mixtures; simple reactions 
leading to activity series, acids, bases and 
salts, laboratory preparations of common sub- 
stances , etc • 


Chemical 


25/26/ 




substances 


(28/29) 


very simply 


and processes 


30/31 





Topic 


Bef. in 
Grid 


Comment 


Physics 




Signs of change from traditional subject 
matter to more modem material are more ob- 
vious here than in biology and chemistry* 

To meet this situation perhaps we ought to 
collect items reflecting traditional treat- 
ments and modern ones as well* For this 
reason subject areas not entirely common to 
all countries have been included exception- 
ally* 


Measurement 


35 


Especially elementary ideas of accuracy, 
approximation; fundamental and derived units 


Motion 

(Kinematics) 


36 




Forces (equil- 
ibrium:, causing 
motion) 


37 




Energy (changes 
and conserva- 
tion) 


39 

t 




Machines (mech- 
anical advantage 
efficiency) 




Mechanics of 
fluids 
(Archimedes 
Principle 
Buoyancy) 


40 




Elementary heat 
(expansion, 
change of 
state) 


41/42 




Light (elemen- 
tary optics) 


44 




Sound (vibrations 
and waves 
simply) 


45 




Elementary 

Electricity 


48/49/ Simple circuits!. Effects of electric cur- 
50 rents (magnetic, heating (chemical) , that is 

qualitative treatment only, not Ohm's law* 
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Target Population (4) (see Bulletin 2 pages 2 and 3) 



Topic 



Ref* in 
Grid 



Comment 



Practice with regard to the earth science 
is so variable that as with Populations 2 
and 3 there is not sufficient common ground 
to justify testing. 



Biological Sciences 




Cells as units 




of structure 


7/8/ 


and function 


7/8/ 


(in more 


9/10 


detail) 




Genetics, practical 




and theoretical 




aspects 


11 


Diversity of 




life, incl. 


12 


classification 




Biology of 


13/14/ 


organisms (in 


15/16 


more detail) 




Natural groups, 




speciation and 


20 


classification 




Evolution 


22 


Chemistry 




Basic facts, 


23/24/ 


methods, 


25 / 26 / 


principles and. 


27 / 28 / 


theory of 


29/30/ 


chemical 


31 


sciences 




Chemistry of 


33 Practice varies considerably from country to 


carbon compounds 


country but it appears desirable at least at 


and of the life 


this preliminary stage of the project to 


processes 


produce some questions on organic chemistry* 
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Topic 


Ref, in 
Grid 


Comment 


Physics 




See general comment on physics for Popula- 
tions 2 and 3* 


Physical measure- 


meats, con- 


35 




stants, methods 
of measuring, 
theoretical 
aspects 


Statics and 


36/37/ 


More emphasis should be placed upon the 


Dynamics 


38/39/ 


experimental and applied aspects than upon 




4o 


the mathematical ones. 


Heat 


bl/kZ 




Kinetic theory 


43 




Light 


44 




Sound and wave 


45/46 




phenomena in 
general 


Spectra 


47 




Electricity and 


48/49/ 


Practical and theoretical aspects including 


Magnetism 


50 


quantitative work 


Electronics 


51 




Atomic physics 


52 
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READING COMPREHENSION RATIONALE 




APPENDIX E 



READING COMPREHENSION RATIQNAT.F. 



Since reading comprehension is an important tool for the under- 
standing of most other subjects, the IEA Council has decided to include 
it in Phase II of IEA. 

The Heading Comprehension Committee considered a number of views 
of reading comprehension and concluded that reading comprehension re- 
garded as instrumental to learning is common across countries • Because 
of the difficulty of handling aesthetic and affective characteristics 
of written material in translation, and because there is a literature 
committee which will include these in their study, the Committee thought 
it best to limit reading comprehension to cognitive aspects. 

It seemed advisable to the Committee to try to reproduce in the 
tests the commonest situations in which reading is used in schools. 

The three main situations appear to be: 

a. Assignment type reading where questions are answered with 
the passage available for reference (III-1-a). 

b. Heading to remember and recall (Ill-l-b). 

c. Directed type reading, in which particular lines of inquiry 
are prescribed, and questions are answered without the 
passage available (III-l-c). 

For the same reason, the Committee thought it best to use para- 
graphs rather than isolated sentences to test reading comprehension. 

Oral reading was ruled out because it would necessitate individual testing. 

A number of related measures could have been used but those chosen 
were thought most likely to cover the dimensions of variation which con- 
tribute to reading comprehension. In this respect, vocabulary seemed an 
obvious choice, but the problem of finding words of comparable levels of 
difficulty from country to country argued in favor of a measure not so 
dependent on word familiarity. A test of word meaning was therefore 
adopted which depends on judgment between pairs of words where the dif- 
ficulty lies in their relationship rather than in their rarity. (HI-2-a) 

The intimate relationship between rate and comprehension persuaded 
the committee that a separate measure of speed of reading was required. 

A device is to be used in which rate will be measured in relation to a 
constant level of difficulty. (HI-2-b) 
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Individual differences in adjustment of rate to the level of 
difficulty suggested a further test in which the degree of adaptation 
of speed of reading to level of reading difficulty could be assessed • 
(III-2-c) 
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A study of reading comprehension aimed at selected school popu- 
lations (II-Target Populations) should enable us to test a number of 
hypotheses. The Committee has drawn up a list of suggested hypotheses 
which is not meant to be exhaustive. Comments, and further suggestions 
are asked for:- when making any comments or suggestions, however, it 
should be borne in mind that comparisons across countries are of pri- 
mary interest in such a survey. 

I. P^POSBD_HYPOTHESK 

1 . Internal structure of the test battery 

a. What are the relationships between the different tests of 
reading comprehension in the different countries and how do these 
relationships change according to the level of education of the 
target populations? 

b. What are the relationships between reading comprehension, 
rate of reading, vocabulary, adjustment of rate of reading, in the 
different countries; how do they change according to the level of 
education of the target populations? 

2. Educational_gractices and _background 

The level of reading comprehension may be influenced by: 

a. methods of instruction at the beginning of reading (i.e., 

analytic, synthetic, mixed). 

b. pre-school instruction in reading. 

c. importance of reading in the curriculum (definite reading 

lessons). 

d. formal training in grammar, in vocabulary. 

e. kinds of reading assignments if any (literary appreciation, 

extensive reading, memorizing poems or texts, etc.). 

f. number of teachers the child has each week. 

g. type and size of school. 

h. type of studies (i.e., general, technical • • •) 
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i. specific subjects studied. 

j. grade. 

k. advance or retardation in studies. 

l. th© grade at which a second or foreign language is introduced. 

3. It may also be_influenced - b^_the_availabilit^_of_reading_fflaterial. 

a. in the school (does the child have textbooks for different 
subjects , is there a school library?) 

b. in the home (books, newspapers, magazines) 

c. in the community (can books, newspapers, magazines be bought 
or borrowed?) 

if. The environment outside the school jfla 2 _have_an_influence_on 

a. Occupation and social standing of the parent (s) 

b. Level of instruction of the parent(s) 

c. Amount of exposure to mass media (radio, TV, films) 

d. Is the language of instruction the child’s first language? 

e. Is the child monolingual? 

5. The interests of the_child_^d_his^attitudes_ma 2 _haye_^_influence 
on'readlng “comprehension. " 

a. Preference for some of the subject-matters (by direct rank- 
ing, by ranking of the different texts of the test, etc.) 

b. Interests in leisure activities 

c. Attitudes toward reading (child's idea of the aims of read- 
ing, why is it important to be able to read, etc.) 

d. Ambitions and aspirations of the child 

e. Attitudes of the child toward school (does he regard it as 
a compulsion) 

6. Study and reading habits_of_the_child_r^y_influence_his_reading 
comprehension • 

(e.g. , finger tracing, vestigial oral reading, etc.) 
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(e.g. , eye sight, hearing, speech defects) 

IX. ?^_POPULATIONS 

1. Th©_mos t_±m^or"taiit populations seem to be at! 

a. Towards the end of primary education (around 10 or 11 ): 

This stage was considered to be important in the testing of read- 
ing comprehension for several reasons. One is that in many countries 
almost all the children are still in school and have followed the 
same course of instruction. Another reason is that it is the stage 
at which, in most countries, children have finished formal training 
in the mechanics of reading. It is also the stage when children, 
entering the secondary school, are expected to be able to read more 
independently. 

b. The pre-university level! This is the best stage at which 
to attempt to assess the overall effect of primary and secondary 
education. At this stage most pupils probably have completed any 
formal instruction or training they will get in reading. In the 
university, they are expected to use extensive reading as one of 
the main tools of study. 

2. £°PHi a £i222_2i22_22 n 2iSj2E22 of interest in such a survey at a 

jr_clea?Iy_an5_cons±s beribX^~across — coxlnbries - ar*e — ^Hose — abT 

a. the level of transition from general to specialized educa- 
tion (about 14-15) and/or 

b. the terminal point of education for many pupils which usually 
coincides with the end of the compulsory education period. 

III. k™s_of_j^tomnt_envisaged 

As it has already been stated, it is intended that the reading com- 
prehension tested will be limited to the communication of ideas and will 
exclude perception of feeling, appreciation of aesthetic quality or purely 
literary devices. Other aspects of comprehension which are excluded are! 

a. the comprehension of other symbolic media, e.g., plans, diagrams, 
charts, graphs, maps, etc. 
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b. listening or aural comprehension - one exception might be 
a short test to be sure that pupils especially at lower 
levels have understood what they are to do (i.e., what the 
task is, how they are to mark the answer sheet, etc.). Any 
testing of purely oral reading is also excluded. 

1. In_view_of_the__above__considerations i _the_followi n g types of 

a. Silent reading of passages continuously available to the 
pupils followed by written questions. 

b. Silent study of a passage which is then removed before the 
questions are asked. 

c. Silent study of a passage after the pupil has been given 
some direction as to the type of information he is expected to 
get from the passage. The passage is to be removed before the 
questions are asked. 

N.B. The questions on the passage will be multiple choice type 
and/or completion type with no choices given (the questions be- 
ing so designed that there is only one right answer). 

2. Sugglementary 

The aim of these is to test diagnostically some of component 
skills of reading comprehension. 

a. A word meaning test 

b. A separate measure of rate of reading is envisaged. It will 
be carried out at a simple comprehension level. 

c. A test of adjustment of the speed of reading to the diffi- 
culty of the material. 
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APPENDIX F-l 



THE E LEMENTS OF CRITICISM AND INTERPRETATION 

^ ^ 

Alan C. Purves 

a study of the ways in which students, teachers, and critics 
a literary work, we have found it necessary to devise some 
describing the critical process, 2 The usual classifications - 
formalist, Aristotelian, Marxist, Freudian - are not adequate because 
| such terms do not really show what happens in a piece of writing, and 

I because few essays fall neatly into any one niche (and they should not, 

I for were they to fall so, the end of criticism as a human encounter with 

literature would be at hand). One can more clearly perform the task 
i by identifying the intellectual operations that constitute a piece of 

| critical and interpretive writing, 

| These operations would be discrete, dements, as it were, of criticism 

| and interpretation, which may be combined in many ways to form a 

critical method. In fact, the experienced reader might employ nearly 
I all of the lements - at least subconsciously - every time he confronts 

| a new work. Some he ignores, some he emphasizes, some he subordinates 

( to others, as he goes about the act of reading and writing about his 

reading, 

§ 

! As one looks at the list below, one must realize that within the 

four categories - or between them, for that matter - there is no rigid 
| hierarchy. Certain sub-groupings represent the idiosyncracy of the 

J writer, yet the terms themselves, as well as their placement within 

categories, represent the consensus of our advisers, and, I hope, are 
neutral. What value one is to assign to the use of any element or 
I combination of elements is beyond the purview of this study. 

| — — 

| 

A brief version of this paper appeared as an interim report of The 
Horace Mann-Lincoln Institute of School Experimentation, New York 
1965. 
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iii Summary of Critical and Interpretive Modes 

| 

I. Engagement - Involvement 

A. Reaction to the content of the work 

1. Moral reaction 

2. Word associations 

B. Identification of reader with the work 
1. Relation of incidents to those in the reader's life 

C. Redaction of work from a point of view different from that stated 

D. The impressionistic monologue which speaks more of the reader's 
sensibility than of the work 

E. The divergent response 

II. Perception of the Work 
A. Analysis of the pattern of order of the word 

1. Language 

a. Grammar and typography 

b. Syntactic and rhythmic patterns 

c. Diction 

2. Literary devices 

a. Rhetorical devices 

b. Imagery or image patterns 

c. Larger literary devices 

3. Content and structure 

a. Narrative time or action 

b. Plot structural analysis 

c. Character analysis 

d. Character relationships 

e. Setting 

4. Tone or modd 

a. Establishment of the writer's or speaker's tone or point 
of view on the mood of the work 

b. Pace of the work 

c. Mask 

5. Total structure or Gestalt of the work 

6. Rhetoric of the work 

a. Relation of technique to meaning or effect 

b. Presentational elements (drama) 

7. Typological figures 

a. Common symbols 

b. Allegorical relationships 
B. Classification of the work 

1. Generic classification of any part 

2. Traditional 

3. Contextual 

a. With the author's other works 

b. Biographical 

c. Intentional 

d. Historical 

e. Intellectual 

III. Interpretation of the Work 
A. Interpretation of parts (may follow any mode in B below) 
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1. Interpretation of a passage of the work as a key to the 
whole work 

2. Derivation of specific svnbols 

3. Conjecture about the past and present in the work 
B. Interpretation of the whole 

1. Mimetic 

a. Social 

b. Political 

c. Historical 

d. Moral 

e. Psychological 

2. Typological 

a. Psychological 

b. Theological 

c. Political 

d. Philosophical 

e. Archtypal 

f. Aesthetic 

3. Moral 

a. Ethical 

b. Poltical or governmental 

c. Social or sociological 

d. Psychological 




v. 



IV. Evaluation of the Work or any Part of the Work 

A. Affective 

B. Formal 

C. Coherence 

D. Generic 

E. Traditional 

F. Intentional 

G. Mimetic plausibility 

H. Thematic Plausibility 

I. Symbolic Plausibility 

J. Moral acceptability 

K. Multifariousness 

L. Rhetorical 



iii The Categories (Engagement - Involvement, Perception, Interpretation, 

Evaluation) 

The establishment of these categories was the major work of our 
advisers, and represents a general picture of the intellectual operations 
one performs with literature. Engagement "Involvement is the best term 
we could find to define the combined acts of entering into a literary 
work and experiencing it or surrendering to it. Alternative terms like 
response or assimilation seem over-restrictive. To some extent, engage- 
ment-involvement is of a different order from the other three categories 
in that it encompasses a process and in that it is often not recognized 
as a part of the critical tool-kit. Yet some excellent criticism has 
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arisen £rom engagement -involvement, and although it has fallen from 
favor among many critics it appears a necessary adjustment to the 
reading o£ literature and is certainly not to be discussed as irrelevant. 
A: times, in fact, one is hard put to distinguish a statement that re- 
lates a work to the reader's life from an interpretation of the work 
(the di££erence may simply be one of generalization). 

The second category, perception, is almost self-explanatory: 
it encompasses both those parts of a work that a person looks at and 
the lens through which he looks at them. The first we might call an 
inductive mode o£ vision, the second deductive; the first arises from 
a point-of-view that comprehension comes from a regard to detail, the 
second £rom a regard to larger structure, a pattern of literature 
rather than of the literary work. _ 

Interpretation , not unlike engagement-involvement, defines that 
area in which the work and the readers interact. It is the act by 
which the reader, who has established the "otherness" of the work, 
seeks to connect it with himself and his world. He says what the 
work means, which is itself a generalization, and then proceeds to 
put it in an even more general scheme. The process is one of abstrac- 
tion or analogy and arises from a perception of the work as a whole 
that exists in a larger, non-literary universe. At a simpler level, 
it is also the drawing of inferences from the work, or from a part 
of the work. Interpretation ranges from the simplest conjecture about 
the £uture beyond the work through an attempt to see the work in re- 
lation to the known world to an attempt to assimilate the work into 
a larger moral order. Each mode of interpretation then arises from 
a particular view of the function of literature in society: as an 
imitation of life, as a distillation of reality, or as a medium of 
judgment about life. 

Evaluation , the last category, encompasses the process of judging 
the work by some criterion, personal or objective. In this category, 
the order of elements roughly parallels the order of the elements in 
the other categories. One's judgment of a work follows from one's 
engagement -involvement, perception, or interpretation of the work; 
there would seem to be no other premise of taste. 

These four categories exist as the general framework of the elements, 
but they exist in no particular order or procedure. Any one way precedes 
any other in time or in logic: one's reaction may follow from or produce 
one's interpretation or perception; an evaluation may be instantaneous 
or the result of a rigid examination of the text. In fact, an attempt 
to impose any rigid structure on these interwoven mental acts would be 
useless, because we do not know and can probably never know just how 
the mind of a reader works. 

Before proceeding to define each of the elements, I should like 
to «make several points about them. First, some of the words or phrases 
listed are not in themselves elements, but simply terms used to group 
elements: i.e. II A (Analysis of the pattern or order of the work); 
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II A 1 (Language) ; II A 2 (Literary devices) ; II A 3 (Content and 
Structure) ; II A 4 (Tone or mood) ; II A 6 (Rhetoric of the work) ; II A 7 
(Typological figures) ; III A (Interpretation of parts) : and IIT B (In- 
terpretation of the Whole) 0 Second, if these elements are used as a 
basis for analyzing a piece of writing, some sentences would not be 
included - sentences which are included for rhetorical effect or 
sentences which cite criteria for literature but do not actually judge 
the work. Third, for convenience the elements of II A (Analysis) are 
listed by the thing perceived or analyzed, and those of valuation by the 
criteria. The last, and most important, point is that although many of 
the elements could be subdivided, I think such a task would be fruitless 
and would render such a classification fruitless. There could be sub- 
divisions based on genre or on schools of interpretation, but they would 
tell us little more about the cast of the writer's mind. It is less 
important, I think, to know whether a critic's political interpretation 
is rightist or leftist than to know that the critic sees the work as 
the reflection of a political idea. If anyone wishes to atomize the 
elements, he is free to do so. 

iv The Elements of Engagement-Tnvolvement 

The expression of each of these elements seems to reflect more of 
the reader than of the work. That such reflection is not unimportant 
to the literary experience, few would deny, nor would they that it is 
related by some alchemy to the other parts of the critical and inter- 
pretive act. 

I A Reaction to the content of the work incorporates diversity: interest, 
boredom, horror, amusement. 

I A 1 On another level, reaction may take the focus of such a moral 
judgment or conjecture about a character or event as seems to arise 
from a failure to distinguish between art and life. 

I A 2 Word associations are a minor form of this reaction, whereby a 
word in a work provokes such a reaction from the reader that his attitude 
toward the text is admittedly prejudiced. 

I B Identification of the reader with the work is close to what has 
been called the vicarious experience. A reader might say, "I felt as 
if I were there," or otherwise indicate his submission. 

I B 1 Relation of incidents to those in the reader's life is a form of 
identification, which, in an essay, often leads to a discussion of the 
reader's life, not the work. 

I C Redaction refers to the process whereby a reader second-guesses 
the author either by changing the point- of -view or rearranging some 
aspects of the work. Although the element might be associated with 
evaluation, a dissatisfaction with the work, it seems more clearly 
an off-shoot of a person's experience of the work and dissatisfaction 
with that experience. 
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1 D The impressionistic monologue, not unlike I B, is the expression 
of the reader* s personality more than of his view of the work. At 
times it stems from an attempt to transmute the work into a second work of 
art - the first seen through the lens of a heightened sensibility 
more often it embodies the reader's fixing upon an idea, event, or mood, 
divorcing it from the work, and expanding upon it. 

I E The divergent response is an extreme form of I B. If someone were 
asked to discuss a poem and instead wrote another poem, or an essay 

on his personal problems, such a response would be divergent more than 
impressionistic • 

v. The Elements of Perception 

II A 1 a Grammar and typography include such phenomena as odd devices 
of punctuation, grammatical errors, or visual effects (the odd 
spacing or indentation of poetic lines) . 

II A 1 b Syntactic and rhythmic patterns include meter, cadence in 
prose, phonetic patterns, and other stylistic devices involving the 
ordering of language. 

II A 1 c Diction encompasses the author's choice of words, whether 
level of usage, lexical oddity, or even semantic ambiguity. 
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II A 2 Although some might consider these elements just of language, 
many would separate them because they connect language and content and 
are not simply linguistic considerations. These elements are to be 
differentiated from II A 6, because the former remain at the level of 
a catalog. 



II A 2 a Rhetorical devices include such traditional figures as 
metaphor, metonymy, and zeugma. 



II A 2 b Imagery refers to those striking, repeated, or characteristic 
references to a particular milieu that an author may make. 



II A 2 c Larger literary devices include diologue, description, 
narration, and others which are not generic but which constitute 
parts of a work. 



II A 3 a Narrative line or action refers to the order of events as 
the author presents it. In an essay, this element is signalled by a 
repetition or paraphrase of the text. 



II A 3 b Plot or structural analysis differs from 3a in that here the 
reader is dealing with the author's ordering of events rather than the 
events themselves j he perceives a pattern in the text. 



II A 3 c Character analysis includes the attempt to explain motivations 
or otherwise describe a character. It remains at the level of per- ^ 
ception as long as the reader does not conjecture about the character s 
motivation or past where these are not given. 
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XI A 3 d Character relationships involves the discussion of static 
relationships - the positions that a character might have vis-a-vis 
another. As soon as the relationship becomes dynamic, the perception 
would be of plot. 

II A 3 e Setting refers to the perception of the locale, either 
in itself or in relation to narrative or character. 

II A 4 a Establishment of the author's or speaker's tone or the mood 
of the work may seem close to reaction, yet many critics would agree 
that despite the difficulties of describing tone and mood objectively, 
the attempt is to be objective. This element will often be supported by 
elements II A 1 or II A 2 and their appearance might well serve as 
evidence of objective intent. 

II A 4 b Pace is even less easy to establish than tone; "the work 
moves quickly" is close to impressionism, yet the same criterion as 
for II 3 A a applies. 

II A 4 c Mask involves II A 4 a, but assumes a duality in the work: 
that the narrator and the author are not the same - or that they are. 

II A 5 Total structure or Gestalt of the work refers to an attempt 
to describe the whole work, often in terms of a spacial metaphor ("it 
is circular"; "it turns on itself") or even in terms of a musical 
metaphor ("a rondo form"). 

II A 6 a Rhetoric of the work: the relation of technique to meaning 
or effect is perhaps a step beyond simple perception; it is the attempt 
to relate perception to the meaning, to explain parts of the work in 
terms of the whole, or to relate minor with major aspects of the work. 
This activity is based on the reader's assumption that the work is 
organic, that everything in the work is designed to aid in creating 
the work' s meaning and effect. 

II A 6 b Presentational elements refers to the aspect of the rhetoric 
peculiar to drama, the relation of the visual to the aural. It follows 
from the reader's attempt to visualize the play. 

II A 7 a Typological figures: common symbols identifies the perception 
of those common symbolic referents in literature that cannot be taken 
as anything but symbolic. The perception of these seems close to 
typological interpretation, but one can distinguish the two as follows: 
Blake's T.anfo in "Little Lamb, who made thee," cannot but be identified 
with Christ and the Christian. His Tyger, however, is a tiger and 
although it can be interpreted as power, or Christ, or truth, it remains 

a tiger. 

II A 7 a Allegorical relationships, like common symbols, refers to 
the reading of those words, like the beast fables or Pilgrim* 8 Progress, 
in which the allegorical level is readily available to any reader. Some 
works of Kafka, too, demand this sort of perception, even though the 
specific allegorical reference is debatable. 
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XI B Classification of the work as a general category refers to those 
perceptual acts which see one work as a part of literary history, and 
as a specific category to those acts which see the work as related to 
a literary prototype. 

II B 1 Generic cla sification of any part defines the acts of typing 
a character (the servant of Roman Comedy) , of making analogies between 
aspects of works, or by typifying an aspect of the work. 

II B 2 A traditional is like a genuine classification, differing only 
in that the work is seen as a point on a contuuum within the genre not 
merely as an object to be put in a drawer. The reader may go on to show 
the work's deviation from the tradition. 

II B 3 This group of elements involves the act of seeing the work as 
a product of its author and of a point in time. The act is in its 
simple form classificatory, but the reader may well go beyond that act 
to a discussion of the relation of the work to its matrix. The sub- 
elements list the predominant focus of contextual classification: with 
the rest of the author's work (a), with the author's life (b) , with 
the author's expressed or inferred purpose (c) , with the events at the 
time of the work's creation (d) , and with the history of ideas or a 
philosophic or religious point-of-view associated with the author or 
his times (e). 

vi The Elements of Interpretation 

HI A Interpretation of parts is the element of interpretation in 
which the reader singles out a particular Segment of a work and proceeds 
to generalize about it, although he may make no generalization about 
the whole work. 



Ill A 1 Interpretation of a passage as a key resembles III A, but the J 

reader here asserts the passage to be a reduction of the whole work. J 



III A 2 Derivation of specific symbols includes that process by 
which a reader invests an object with typological significance (see the 
explanation of II A 7). 

Ill A 3 Conjecture about the past and present of a work is the form 
of interpretation closest to I A (Reaction), but can be differentiated 
thus. Interpretative conjecture seeks to ascertain motive or thought 
within the time limit of the work, and from data given in the work. 

The conjecture of reaction seeks to continue the work beyond the limits 
given by the author - to wonder, for instance, whether Hamlet will go to 
heaven or whether a couple really will "live happily ever after." 
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III B 1 Mimetic interpretation, like the rest of the elements in III, 
refers to a mode of interpretation. This mode imputes a reader's 
seeing the work as a mirror of the world, either generally or in one 
of the five specific ways. The reader says, "This is the way the world 
is. ' If the work is an heterocosm, it is one which is closely connected 
to the world around the reader. 
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Ill B 1 a Social mimetic interpretation refers to a world seen as 
the interaction of types of people, classes, or societies, 

III Bib Political mimetic interpretation refers to a world seen as 
a power structure made up of the governors and the governed. 

Ill B 1 c Historical mimetic interpretation refers to a world seen 
at a specific time in the past or at recurring similar times (the 
fall of empires). 

Ill Bid Moral mimetic interpretation refers to a world seen as made 
up of ethical forces. 

Ill B 1 e Psychological mimetic interpretation refers to a world made 
up of conflicts within the mind of individual man. 

Ill C 2 Typological interpretation sees the work not as a mirror, but 
as an abstract or a highly generalized pattern of the world; the work 
becomes at least typical, at most allegorical or a metaphor of some 
pattern of existence. The patterns may be psychological (a) , theological 
(b) , political (c) , philosophical - a systematic philosophy - (d) , 
archtypal - a pattern of repeated figures or movements like fertility, 
harvest, sacrifice - (e) , or aesthetic - a pattern in which all art is 
about the artist. 

Ill C 3 Moral interpretation sees the work as a proclamation about 
how the world should be; it sees the author as hortatory - either 
overtly or covertly so. The author is seen as taking a stand, either 
ethical (a), political (b) , social (c) , or psychological (d). 

vii The Elements of Evaluation 



These elements are cast in the form of criteria, either for a 
subjective or objective appraisal of the work. 

IV A Affective evaluation uses the criterion of emotional appeal. 
Either the work succeeds or fails in moving the reader, or in presenting 
its dominant emotion with sufficient intensity. 



IV B Formal evaluation uses the criterion of aesthetic evaluation. The 
work may or may not fulfill its function, succeed or not to use all 
of its parts coherently like a good ballet. Often the untrained reader 
will say that he does not "like the looks" of the work; this, too, 
expresses a formal criterion, albeit weakly. 




IV C Rhetorical evaluation uses the criterion of effective use of 
form to present and enhance content. It judges work by adequacy of 
parts to the whole. 



IV D Generic evaluation uses the criterion in which the reader uses 
the abstract notion of genre to judge a work. The best example is 
the critic who says, "It's a bad form; it does nett rhyme." 



IV E Traditional evaluation uses a criterion akin to that of genre, 
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but less rigid; it judges the work according to the history of its type 
in form or content* Its criterion , then, is flexible* since tradition 
is, within limits, continually modified. 

IV F Intentional evaluation uses the criterion of the author* s 
expressed or inferred intention. 

IV G Mimetic plausibility is the criterion of surface veracity, or, 
on a more rarified level, of the adequacy of the author’s vision of 
life when compared with the reader’s vision. 

IV H Thematic plausib ility is the criterion of sufficiency of meaning. 
It asks of the work that it have an import equal to the reader* 8 set 
of values. It differs from IV G in that it asks, "Does the author 
think of life as 1 do (or as most do)," and IV G asks. Does the author 
represent a world in which I can believe?* 

IV I Symbolic plausibility is the criterion of congruence of patterns. 
It asks of the work that its abstraction of the world accord with the 
reader’s abstraction. This criterion might also be one of the rhetoric 
of symbols: the reader might see the work as reaching for a symbolic 
structure and succeeding or failing in achieving it. 

IV J Moral acceptability is the criterion of lessons. "Does the 
work teach what I consider morally correct?" asks the reader. 

IV K Multifariousness is the criterion of levels. The reader asks 
of the work that it be interpreted in many ways and judges accordingly. 

viii Conclusion 

This schema, I hope, will be of use to critics and teachers. By 
separating the elements of criticism and interpretation, one can 
possibly regroup them; one can observe what it is he or his students 
do and seek to rectify omissions; one can even see the relationships 
which exist between certain works or genres and certain elements. 
Finally, the literary historian can use the schema to examine the 
critical temper of a past age and, perhaps, can draw inferences about 
the characteristic ways of approaching reality that are indicated 
by the characteristic ways of approaching the art work. It is this 
purpose more than any other that provoked the schema; yet a tool of 
intellectual history may became a tool of education. 
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Appendix B KEY FOR SCORING 



I. ENGAGEMENT 

01 Rhetorical filler 

02 Reaction to content 

03 Word associations 

04 Relation of incidents 

05 Identification with work 

06 Redaction of work 

07 Impressionistic monologue 

08 Divergent response 

II. PERCEPTIO N 

09 Narrative line 

10 Plot analysis 

11 Character analysis 

12 Character relationships 

13 Setting 

14 Language 

15 Diction 

16 Literary or rhetorical devices 

17 Imagery 

18 Tone 

19 Mask, point of view 

20 Total structure (Gestalt) 

21 Rhetoric (Form & content) 

22 Presentational (Drema) 

23 Classification of work (as whole) 
34 Generic (part of work) 

25 Traditional 

26 with author's work 

27 Biographic 1 

28 Intentional 

29 Historical 

30 Intellectual 

31 Common symbols 

32 Symbolic relationships 

65 Typographical -grammatical 

66 Reading comprehension 

67 Perceptual general 



III. INTERPRETATION 

33 Passage as key 

34 Derivation of symbols 

35 Mimetic 

36 Mimetic-social 

37 -Political 

38 -Historical 

39 -Moral 

40 -Psychological 

41 Typological 

42 Typological-Psychological 

43 -Theological 

44 -Political 

45 -Philosophical 

46 -Archetypal 

47 -Aesthetic 

48 Moral -Ethical 

49 ^Political 

50 -Social 

51 -Psychological 

68 Interpretive General 

69 Interpretation of parts 

70 Conjecture 

IV. EVALUATION 

Affective 

53 Formal 

54 Generic 

53 Traditional 

56 Intentional 

57 Mimetic plausibility 

58 Thematic plausibility 

59 Symbolic plausibility 

60 Moral acceptability 

61 Multifariousness 

62 Rhetorical 

63 Unspecified 

71 Citation of Criteria (Art) 



52 



V. Miscellaneous 
64 Unclassifiable general 

Numbers 65, 69, and 70 are sub-categories of the elements that appear 
frequently enough to become categories in their own right. Numbers 01, 
63, 64, 66, 67, 68, and 71 were added to classify sentences that appear 
in many essays, but that are not elements of criticism or interpretation. 
It is expected that as the study progresses certain elements will 
receive greater or less emphasis and that certain elements will be 
collapsed. 
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Appendix C. STATUS OF PROJECT 

As of the summer of 1965* an outline of the elements of criticism 
and interpretation has been developed, together with a scoring sheet 
for student essays. 

Sample essays by approximately 100 American, 50 Belgian, and 50 
German students, age 13 and 17, all writing on a short story have been 
gathered. The essays will be scored and the data analyzed by the end 
of the summer. This work should test inter-reader reliability and should 
give preliminary data to give tentative support to Hypothesis II and III 
a (the major hypothesis of the project). 

In addition, preliminary work on the analysis and comparison of the 
oral and written responses by a small group of students to a variety of 
literary works will have been completed. 

An anlytic scheme has been developed in cooperation with Teachers 
College Computer Center. Results of a first analysis of the data 
obtained from the papers above will be presented to the participants 
in the IEA meeting at Hamburg, August 30, 1965. 
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APPENDIX F-2 



LITERATURE RATIONALE 



The general purpose of this- otudy is to shed light on the rela- 
tion between the humanistic aims of literary education, upon which we 
believe there would be broad agreement, and both school policies and 
practices and the place of literature in the cultures of different nations. 

This we propose to do by examining the performance in, and at- 
titudes towards, literature of students at the end of compulsory educa- 
tion and at the end of pre-university training. 

The vital educational ’’unit" in literature teaching might be 
said to be the individual student* s reaction to an individual work of 
literature, since, however, this is an imponderable, this study will 
focus upon his expressed, or formulated, responses. We expect that 
there will be characteristic national and cross-national patterns in 
such responses and that these will be related to aspects of the stu- 
dentfs literary education, 

HYPOTHESIS^! 

Characteristic patterns of expressed response will be closely 
related to knowledge about and attitudes towards literature. 

There are various outcomes of literary education, such as types 
cf response to literature, knowledge about literature, attitudes towards 
literature. These are given in detail dft page F-16.~ These outcomes do 
not combine to form a single unit of achievement, but they are related 
to each other in a variety of ways and degrees. An often made comment 
is that "The more a student knows about literature, the less is his de- 
sire to read a book," 

HYPOTHESia_II 

The various outcomes of literary education are related to, but 
not identical with each other. 

The principal education officers commonly spend much effort on 
official statements of aims in the various parts of the curriculum. 

These statements serve a double purpose. On the one hand they explain 
the purpose of the various subjects to the public, and on the other 
hand, they seek to influence teaching in the classroom. For literature 
the latter purpose is often not well served. This study will seek 



Culture throughout refers not to anthropological culture but to 
aesthetic culture. 
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evidence on this particular point and may enable countries to learn from 
each other what are the most effective ways of setting out objectives. 

HYPOTHESIS_III 

International differences in the stated aims of literary educa- 
tion are related to the place of literature in a nation’s culture and 
practices of literary education. 

hypothesis^iii-a 

International differences in the outcomes of literary education 
are more closely related to the practices of literary education than to 
the official statement of its aims. 

In the mathematics study it was found that influences outside 
school were much stronger than had been anticipated. In the case of 
literature teaching the outside influences, particularly the cultural 
aspects, might be expected to be highly important. 

HYPOTHESIS_IV 

Within and between nations, outcomes of literary education will 
vary in relation to social, cultural and individual characteristics of 
students. 



Finally, education in literature is related to the differences 
or the similarities between an elite literary culture and a popular 
literary culture. There may be a highly ambiguous and uneasy relation- 
ship between these two cultures (e.g., Macbeth and the comic newspaper) 
and this tension influences the aims, practices and outcomes of liter- 
ary education, particularly the attitudes of the teacher and the stu- 
dent towards both the elite and the popular culture. While we do not 
think any simple solution of the problem is to hand, we believe the pro- 
vision of evidence over a wide range of situations may help towards an 
ultimate solution. 

HYPOTHESIS_V 

Internationally, the difference between elite and popular cul- 
ture as represented in the schools is related to the place of each cul- 
ture in the society as a whole. 

The ISA literature committee has tentatively developed instru- 
ments to measure the various aspects of the hypotheses. Some of these 
instruments exist in draft form; some have been specified and we think 
are capable of being developed; all of them we would like to be subject 
to the constant scrutiny of the national committees. Drafts will be 
sent to the committees during the months ahead for comments, suggestions 
for exclusion, revision and addition. Some of them are: 
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a. A scheme for content analysis of written responses. This 
has been substantially developed. 

b. Literary preference measures. 

c. Literary attitude inventory. 

d. Measures of knowledge and application of literary terminology. 

e. Measures of preferred responses to literature. 

f. Measures of knowledge of allusion to national and cross- 
national cultural background (e.g., myth and legend). 

g. Measures of familiarity with major literary texts, both 
national and cross-national. 

h. Measures of reading habits. 

In addition to the stated hypotheses, the IEA literature com- 
mittee has developed many tentative sub-hypotheses and operating hy- 
potheses. These will be the subject of consultation with the national 
committees during the next few months. 
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LITERATURE 

INSTRUCTIONS TO NATIONAL CENTERS 



We would like to obtain from the National Centers cooperation 
in the following ways; 

1. The appointment of a National Committee in Literature which 
would include persons who have, among other areas of competence! knowl— 
edge about the literary criticism and theory of the nation; familiarity 
with teacher training practices; familiarity with the various school 
practices in literature; and familiarity with social science research 
methods. 



2. A review by the National Center of the document entitled 
"The Goals of Literary Teaching" (attached to this document as Appendix 
which sets out a summary of educational objectives in literature as to 
their clarity and accuracy, including suggestions for revision, addi- 
tions and an Indication of the relative importance of the objectives 
and including an indication on the enclosed grid the national emphasis 
of various cells. Queries about the grid are expected; we anticipate 
that it will take three weeks for queries to be made and answered* 

3. A criticism by the Center of the major hypotheses of the 
study as presented in the "Rationale" document, and suggestions as to 
additional major hypotheses. 

4. The submission of copies of official guides for instruction 
in literature (and/or other official documents pertaining to education 
in literature), copies of national or regional examinations in litera- 
ture and any explanatory materials for these exams, titles of the one 
or two most widely used textbooks in literature (if such exist), and 
titles of major books on pedagogy in literature. (NOTE:, the IEA com- 
mittee can read documents in English, French or German) • 



LITERATURE 



THE GOALS OF LITERARY TEACHING 

The study seeks to describe and to measure achievement in the study of 
literature. Such description and measurement will be concerned with 
the following educational goals and sub-goals: 

I. Knowledge_about_literature 

a, knowledge of specific literary texts 

b, knowledge and application of the terminology of liter- 
ary study „ 

c, knowledge of the cultural heritage (mythology and prime 

sources of literary allusion) 

d, knowledge of literary history (This sub-goal will not 
be investigated to any great extent in this study be- 
cause of extreme national and sub-national differences 
in the particulars of its literary history and the em- 
phases which literary history receives. National cen- 
ters are invited to make appropriate case studies with 
regard to this goal,) 

II, Development of Response^ to _Literary_Works 

Response is defined as the result of interaction between the in- 
dividual and the literary work he reads. Such results, of course, may 
be cognitive or affective or both. One of its exponents will be desire 
to continue reading that work. Response is defined variously as appre- 
ciation, taste, or enjoyment. Although it is indescribable in a direc 
sense, it can be described indirectly through the observation of the 
works preferred by the individual. The educational goal is generally 
the development of the threshold of response to include more difficult 
and '’better” literary works. 

Ill . Development M Of_a__Pattern_of_Expressed_Resgonse 

When an individual reports on his response to select parts of 
that response which he thinks important to communicate. The expressed 
response exists in a pattern which can be described f and^ in respect to 
that description measured as to achievement. The means of describing 
the pattern of expressed response is contained in "The Elements of Writ- 
ing about Literature.”* A major educational goal is the development of 
a consistent pattern or series of patterns of expressed response 
that the student will analyze works in a consistent fashion, that he will 
evaluate literary works in a particular way, or that he will view each 
work in its historical context). 



•This document may be obtained directly from: Dr. Alan C. Purves 

Department of English, University of Illinois, Urbana, Illinois 61801 



IV. The Internalization of Positive_Attitudes_Towards Literature 
as~£rTOo![e“an<rthe“AppIIcatIorT^^ " 

gard"to”the”Work”In”^eneraI 

a. The development of the habit of reading 

b. The development of specific positive attitudes towards 
literature and the writer 

c. The transfer of expressed response patterns to non- 
literary events 

These goals may be expressed in terms of the grid on the next 
page. We would appreciate your indicating the emphasis in your curricu- 
lum on the cells of the grid in the following manners 

0 no emphasis 

1 a little emphasis 

2 minor emphasis 

3 major emphasis 



NOTES 






Content 

refers to both whole works and extracts 

kiter^y.histor^ refers to the facts of authorship, text, intellectual 
background, and other such information. Behavior with respect 
to this aspect of content are less important to this study 
than behavior with respect to other parts of the content for 
the reasons stated earlier. 

refers to the terminology and systems of literary 
criticism and theory. 

52liH£2i-.i n £2 r E2^i22 refers to the corpus of mythologic, epic, and 

folk literature of a culture. Parts of this corpus become a 
symbol system in the literature of the culture. 



Behavior 

—22^22—— —2 to the ability to respond to literature and is a major 

goal, but not one that is measurable except as shown through 
preference, expression, and consistent pattern of expression. 

P2Y2l2E-2-2222i2£222LE2i£222-2£- p E2£ e E22222 refers to the development 
of a consIstent"pattern”of"cKoice“of one type of literary work 
over another. 



ExBress_a_resg 2 nse_t 2 may take a variety of forms - verbal and mn- verbal. 

^2222-2-222222 t£2i_£22 p 2222_£2 refers to the habit of expressing a 
preferred type"of "response to a large number of works, 

-2-222»-22 ?;2222;£ e 2-X2i22-222£22}2 are a PPlicable to literature as 

as a whole incXuding texts, history, theory, and cultural in- 
formation. 
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LITERATURE CROSSOVER STUDY 

1. Introduction 

Now that the I.E.A. group is engaged in an investigation of 
sever al new subject areas (Science, Reading Comprehension! Litera- 
ture, Civics, French end English as Foreign Languages), many new 
problems have been encountered. Within each subject area an inter- 
national committee of experts has been meeting for the last two 
years to devise a battery of international achievement tests in their 
particular field. This work is now nearing completion, although in 
some of the subject areas many questions are as yet unanswered. 

One of the international committees, under the chairmanship of 
Professor Arthur W. Foshay of Columbia University, New York, has 
been concerned with constructing instruments to measure the under- 
standing and appreciation of Literature, One important conclusion 
that they have reached is that there are no specific literary works 
which would form universal content area. Neither Shakespeare nor 
Dante, nor indeed any other writer, can be used as a basis for test 
questions in every I.E.A, participating country. The choice then 
seems to lie between allowing each participating country to devise 
its own set of tests, or to introduce examples of literary form 
which are new to all the students in the study, A common set of 
questions on these new pieces of literature (common in the sense 
that they will be the same in each country) can then be put to the 
students. The literary forms that were considered were poems or 
brief sections of prose of about 1,000 words in length. Much disaus- 
sion then centered around whether to use multiple-choice or open- 
ended test items. The point at issue can be illustrated by an 
example: 

a) In lines 1 to 22 John Deweek has many feelings about what 
he sees at the fountain. Which of the following best 
summarizes his feelings? 

(A) He feels that he is separated from other people and that 
he has lost his youth. 

(B) He feels that his heroic past has been forgotten by 
others • 

(C) He feels that the park is unattractive when there are 
people in it. 

(D) He senses that he can be understood better by people 
his own age. 
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b) How does he feel about what he sees at the fountain (lines 1 
to 22)7 



The first form of the question can be machine-scored objectively, 
reliably and cheaply# The second form requires the subjective judgment 
of one or more human graders and tends to be more expensive and less 
reliable. However, the issue of validity was raised by the Committee. 

Some of them believed very strongly that only the second form of the 
item could hope to measure the type of instrument in which they were 
in teres ted. Despite the added inconvenience and expense involved, they 
felt that a test consisting of open-ended items was the only one that 
could be of real value# Of course the crucial consideration here was 
the definition of the achievement that the Committee wished to measure. 

This was difficult to specify. However, it was found to be possible 
to agree on the relevance of a certain type of open-ended question. It 
was proposed by some members of the Committee that carefully designed 
multiple-choice questions might measure the same dimension and that 
therefore these multiple-choice questions might replace those of the 
open-ended variety in the final instrument. To test this proposal, and 
also to investigate some of the problems in putting together an inter- 
national test of literary understanding, two small studies were carried 
out. These are reported below: 

2 . 

a) Design. Two short poems were chosen. These were "Two At Norfolk" 
by Wallace Stevens, and "Ample Make This Bed" by Emily Dickinson. 
Each poem deals with the subject of death. Two tests, one in 
multiple-choice format and one with open-ended questions, were 
constructed for each poem. Each test required 45 minutes to one 
hour to complete. A sample of students was drawn and each took 
two of the tests. The first test was always on the poem "Two 

At Norfolk," end the second on the poem "Ample Make This Bed." 

Four pairings of tests were possible for this, and they defined 
the four treatment groups. The data were subsequently analysed 
for internal consistency within the tests and correlations be« ^ 
tween tests on the two different poems. 

fj 

V 

b) Samgle. The sample for this study consisted of 275 students aged 
17 or 18 . All were drawn from three English grammar schools and 
may be considered well above average in general ability. How- 
ever, since they contained substantial numbers of Science special- 
ists, as well as specialists in Literature, it was expected that 
the sample would show wide score variations on a test of Litera- 
ture. Both poems were by American authors, and therefore presumably 



unfamiliar to most of the sample. The students were randomly divided 
into four treatment groups and each took two tests as outlined above. 

TABLE li SAMPLE SIZE AND INTER-POEM CORRELATIONS 





RESPONSE MODE 










Ample Make This Bed 


Two At Norfolk 


N 


r 


Group 1 


Open-ended 


Open-ended 


63 


.240 


Group 2 


Multiple-choice 


Multiple - 
choice 


70 


.212 


Group 3 


Open-ended 


Multiple - 
choice 


73 


.193 


Group 4 


Multiple -choice 


Open-ended 


69 


.275 



c) Results. The sample size and correlations between the two poems are 
sh.own’Tn Table 1, while Table 2 gives more general statistics for the 
four tests. The results for the open-ended tests fire the averaged 
grades assigned by two Independent markers. The grades for these 
markers related with each other to the extent of 0.96* a surprisingly 
high figure which suggests that a very adequate scoring scheme had 
been worked out beforehand. As the markers did not produce sub- 
scores for different parts of the testy it was not possible for these 
tests to estimate their internal:, consistency . 

TABLE 2^- TEST STATISTICS FOR POETRY 



RESPONSE MODE 


POEM 


N Possible Score Mean Score S.D. 


Open-ended 


Ample Make This 


136 


29 


6.83 


4.5 


Bed 










Open-ended 


Two At Norfolk 


132 


31 


8.47 


4.0 


Multiple-choice 


Ample Make This 


139 


12 


8.22 


1.65 


Bed 










Multiple-choice 


Two At Norfolk 


143 


15 


9.26 


2.43 



For the multiple-choice tests it was possible to estimate reliability 
using KR 20. The estimates obtained were* for "Two At Norfolk" R =0.04; 
for "Ample Make This Bed" R = 0.28. 
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In general these results were rather disappointing. Although the 
data provided no evidence to suggest that open-ended and multiple- 
choice tests were measuring fundamental different aspects of abil- 
ity, the uniformly low values of the correlations between the poems 
(whatever the form of the tests) seemed to suggest that the rankings 
obtained for a group of students would be very dependent upon which 
poem was chosen for the test. Clearly one would ©St, in these cir- 
cumstances, wish to allow each individual country within the I.E. A. 
project to nominate its own poem. To some extent these low corre- 
lations may be produced by the rather low values of the test reli- 
abilities, In fact, if we assume that the open-ended tests had in- 
ternal consistencies similar to those for the multiple -choice tests, 
then low correlations between poems would be very largely explained. 
However, in these circumstances, it might be thought that none of 
the tests used provided an adequate measure of literary understand- 
ing, 

3. Prose_Stud2 

a) Design, A cross-over design similar to that employed in the 
pcetry”study was used. Two short stories, "The Man by the Fountain" 
by Georges Hebbelinck and "My Childhood" by Maxim Gorki, formed the 
basis of both multiple-choice and open-ended tests; » Each story was 
approximately 1,000 words in length. The tests were labelled 7t 8, 

9 and 10, Each required up to one hour to complete. Students were 
asked to take two of the tests, one on each of the two short stories. 
Once again the stories were expected to be unfamiliar to the stu- 
dents, Four treatment groups were defined as in the previous study, 

b) Sample, Four separate samples were drawn: 

(i) . 376 Ehglish school children drawn from one grammar and 

three schools. All were in the fourth year of second- 
ary school with an average age of about 14 years 6 months, 

(ii) 289 English students drawn from the Sixth Form of a gram- 
mar school and a Technical College, The mean age is 

l8 years. 

(iii) 338 American High School students selected from the ninth 
grade, and aged about 14. 

(iv) 289 students from the American twelfth grade aged about 

18 . 

(The tests had been designed and constructed for use with two 
populations of students, one aged 14 and the other 18), As it 
was soon apparent that there were considerable differences of 
performance between samples, no pooling of data took place and 
the results for each sample are reported separately. Each sample 
was divided randomly between the four treatment groups. The ac- 
tual numbers assigned to each treatment are given in Table 3* 
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TABLE 3: RESPONSE MODE AND SAMPLE SIZE BY COUNTRY AND AGE 
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RESPONSE MODE 


ENGLAND & 


WALES 


U.S 


.A. 




The Man by My Childhood 

the Fountain 


14 yr. 


18 yr. 


14 yr. 


18 yr* 


Group I 
(7,9) 


oppn-ended open-ended 


92 


66 


80 


66 


Group II 
(8,10) 


multiple-choice multiple- 

choice 


86 


78 


85 


80 


Group III 
(7,10) 


multiple- 

open-ended choice 


90 


85 


86 


62 


Group IV 

(8,9) 


multiple-choice open-ended 


108 


60 


89 


81 



c) Results* The simple test statistics are presented in Table 4* 
It“Is~interesting to note the higher average scores and lower 
standard deviations of the American students on each of the four 
tests. It will, of course, be necessary to wait for the full 
scale E.A. study to determine if this finding holds for truly 
representative samples* 
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Although there are striking similarities in the score distribu- 
tions (mean and standard deviation) of the four tests, this result must 
be regarded as entirely fortuitous. The maximum possible score for the 
open-ended tests is 35 as against 23 for the multiple -choice form. 



t 

I 



1 



Bach of the test booklets 7 and 9 was graded by three markers. 
The three who completed most of the work correlated 0 . 92 , 0.93 and 0.93 
with each other, which, although not as high as the value achieved in 
the poetry study, must be regarded as very satisfactory. One marker 
had to drop out before the end and was replaced for the last 25% of the 
scripts. The substitute marker correlated O 0 88 and 0.90 with the re- 
maining two. 



On this occasion the markers produced sub-scores for different j 

sections of the test and it was thus possible to estimate reliability j 

through internal consistency for all four tests. These estimates are j 

given in Table 5. They appear to be reasonably consistent, although I 

there is some suggestion that reliabilities were higher in England and j 

Wales, and this is no doubt due to the greater variability of these 
samples • 
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TABUS 5: RELIABILITY ESTIMATES (KB 20 ) 



f 









England & Wales. 


u, 


iS.A. 


Test 


Short Story 


Response Mode 


14 yr. 


18 yr. 


14 yr. 


18 yr. 


7 


The Man by the 
Fountain 


open-ended 


.69 


.85 


.52 


.75 


8 


The Man by the 
Fountain 


multiple- 

choice 


.81 


,82 


.68 


.65 


9 


My Childhood 


open-ended 


.79 


.83 


.63 


.62 


10 


My Childhood 


multiple- 

choice 


.83 


.90 


.74 


.78 



TABLE 6i INTER-TEXT CORRELATIONS 



Group 


Response Mode 


England & Wales 


u.s. 


>A. 


The Man by the 
Fountain 


My Childhood 


14 yr. 


18 yr. 


14 yr. 


18 yr. 


Group I 
(7,9) 


open-ended 


open-ended 


• 60 


.43 


.35 


.41 


Group II 
(8,10) 


multiple- 

choice 


multiple- 

choice 


.64 


.66 


.45 


.75 


Group III 
(7,10) 


open-ended 


multiple- 

choice 


.48 


.50 


.58 


.60 


Qroup IV 

(8,9) 


multiple- 

choice 


open-ended 


.46 


.51 


.39 


.49 
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these correlations are not very high, there is no evidence to suggest 
tha^the size of the correlation depends upon the style of response to 
storv -Open-ended" correlates with "open-ended" as well 
as^multiple-choioe" wiS "multiple-choice," or indeed vd^a mixture. 
T« Tnhip 7 these inter-text correlations have been attenuated to ai 
for unreliability in the two measures. The resulting figures are es- 
timates of correlations between the tests under conditions such that 
. . . y. aA refect reliability. These correlations are, of course, 

somewhat higher than in Table 6, but it la la 

gests that after variance due to unreliabilityisr^oved.roughly^alf 
the remainder is shared between the tests and half *9 specific to i 
dividual tests. 

TABLE 7: INTER-TEXT CORRELATIONS ATTE NUATED 

FOR UNP^T-T ABILITY IN THE TWO MEASURES 



Group 



Group I 

(7,9) 



Group II 

( 8 , 10 ) 



( 7 , 10 ) 



Group I V 

(8,9) 



Response 


Mode 


England & Wales 


U.S, 


.A. 


The Man by the « Childhood 
Fountain 


14 yr. 


18 yr. 


14 yr. 


18 yr 


open-ended 


open-ended 


. 

00 

i-* 


.52 


•62 


• 60 


multiple- 

choice 


multiple- 

choice 


.78 


.70 


.63 


1.00 


open-ended 


multiple- 

choice 


•62 


OO 

• 


.91 


.79 


multiple- 


open-ended 


.58 


.62 


0 

VO 

. 


.77 



choice 



4. Conclusions^and^Recommendations 

The results presented on the preceding pages offer ^evidence 
that multiple-choice questions per se measure anything different 
than onen-ended questions per se. It will, be recalled that thi 
was the single most important point to be tested in these studies. 
Although the correlations between multiple-choice and open-ended 
tests are not high, they are not perceptibly lower than are corre- 
lations between multiple-choice tests themselves or between open- 

ended tests. 
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It is possible, however, to distinguish between multiple-choice 
mi d open-ended tests on financial grounds* While the multiple— choice 
responses were scored by computer after they had been punched on 
cards, the open-ended responses were submitted to expert markers. 

The cost of grading one open-ended script (using three markers) 
turned out to be about 40 cents. The computer cost per script was 
about one cent, to which must be added approximately 2 cents for 
key punching. It is clear that the multiple-choice forms of the 
test will prove much cheaper if large numbers of students are in- 
volved. Although in view of the high correlations between the 
markers, relatively little reliability would have been lost by 
using only one marker instead of three, it would not appear pos- 
sible to reduce the cost per script to much below 13 cents, which 
is four times the cost of scoring a script of the multiple-choice 
format. The high inter-marker correlation was no doubt due to the 
very careful briefing that the markers were given before they com- 
menced work. They had an opportunity to examine a selection of 
scripts and to discuss the marking procedures in very great detail. 
This is rather time-consuming, of course, but it does demonstrate 
that with careful preparation a high inter-marker correlation could 
be achieved. With only on® marker there would, of course, be no 
check on the reliability of the grades produced. The results of 
these two studies suggest strongly that only multiple-choice forms 
be used for large-scale testing in this field. 



In each of the two studies here reported, it would seem that at 
least half the non-error variance can be attributed to characteris- 
tics specific to the text under consideration. Less than half the 
total variance can be attributed to a general measure of ability in 
the literary field. This will be an obstacle to the construction 
of instruments designed to provide highly reliable scores of general 
validity. While it would be possible to construct a multiple-choice 
test to measure reliably the understanding of a specific short 
story or poem (by including a sufficiently large number of questions), 
a reliable measure of general ability to understand and interpret 
literature could only be based on responses to a wide selection of 
literary works. It may well prove to be a problem here, since the 
time taken to complete the test depends in large part upon the amount 
of textual material which the student has to master. The finding 
that we get reliabilities of about .73 from tests one hour long is 
frankly discouraging. 



In view of the low correlations between texts, it would seem 
very unwise to try to compare scores made on different texts in 
different countries. If the I.E.A. project is to proceed with 
measures in the field of literary understanding, it seems clear 
that the texts used must be the same in all the countries involved o 
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FRENCH AS A FOREIGN LANGUAGE RATIONALE 



This document is intended to indicate to the National Centers and 
others who may be interested, the overall rationale for attempting to 
study achievement in French as a foreign language on an international 

basis. It consists of four parts: 

I. Design of the study (from a statistical point of view) 

II. Hypotheses 

III. Variables 

IV. Target Populations 

The section on hypotheses and variables will not include a discus- 
sion of criterion variables, since this will come out of the document 
on content analysis intended for the National Centers. It may be assumed, 
however, that the study will eventually involve the development of an 
extensive battery of tests measuring various aspects of achievement in 
French as a second language. 

I. Design of Study 

1. The main interest of the study will lie in the results which 
will be obtained with comprehensive tests of attainment in French. 

These we hope to administer to representative samples of three levels 
of the populations of French learners: The pre-university, the four- 

teen year-olds , and the ten year old levels (for further details of 
these samples/see below). Since the same set of French tests will be 
administered in all participant countries, and since we hope to have 
statistics about the proportions of students studying French at each 
level in each country, we hope as a result of the study to be able to 
estimate for each country the total number of students who know French 
well enough to satisfy certain standards defined by performance on the 
tests. Thus we hope the study will reveal the "yield” of French teach- 
ing in each country. 

2. Apart from determining the "yield" of French teaching, how- 
ever, the study has many other aims mostly relating to factors which 
affect the extent to which French teaching is successful. Several 
considerations have led us to the conclusion that these aims are unlikely 
to be achieved by comparing the overall means, or even the means of 
selected subgroups, across countries. In each country French is an 
optional subject, so it is unlikely that we shall be able to find out 
whether students who take French are representative of the school- 
going population as a whole. For this reason alone it would be quite 
unsafe to assume that the populations of French learners in the partici- 
pant countries are comparable in general ability or in ability to learn 
French. Furthermore, countries vary in cultural background, in native 
language, and in the orthography they employ with the result that the 
task of learning French varies in difficulty from one country to the next 
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| 3. Bearing all this in mind we propose to study in each country 

| the extent to which attainment in French is dependent on a large number 

{ of interesting and important variables, and then go on to compare the 

f influence of these variables across countries. These variables fall 

under seven heads? 

| a. Pupil variables (e.g. sex, number of hours spent in French 

| class) 

I 

b. Parent variables (e.g. interest in and knowledge of French) 

| c. Teacher variables (e.g. extent of professional training 

f in French and in methods of teaching French) 

d. Method and Course content variables (e.g. stage at which 
I each of the four major areas of French learning — listening, speaking, 

reading and writing — is introduced) 

| e. School variables (e.g. Streaming, size of class) 

I 

| 

j Besides these variations within countries, there are variations 

amongst countries which must be studied for the purpose of comparing 
across countries the effect of the variables already mentioned. 

i f. Mother tongue variables (e.g. the extent to which the 

orthography, phonological, syntactic and lexical systems differ from 
j French) 

g. Country variables (e.g. proximity to or remoteness from 
- a French-speaking community) 

\ 

It is reasonable to hypothesize that each of these variables 
contributes to some extent to the score which children obtain in French 
tests, though of course, the study will determine whether in fact they 
do contribute and if so, to what extent. 

4. In a study comprising so large a number of variables the most 
convenient way to tease information from the data is regression analysis. 
This will involve calculating for each country separately the regres- 
sion of French scores on the '’predictive” variables (i.e. variables "a" 
to "g".) The resulting regression coefficients will indicate the accuracy 
with which French scores can be predicted, granted that one knew the 
predictor measures. Furthermore, each coefficient will indicate the 
predictive power of a particular predictor measure independent of all 
the other predictor measures. This in effect is the equivalent of its 
predictive power when all the other variables are held constant. 

Once the regression analysis for each country has been completed 
it will be possible to compare regressions across countries. If 
regressions are constant, the study will have yielded information of 
great importance in coming to a further decision: whether the relation- 

ship between a predictor variable and French scores is a causal one or not. 
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If the regressor is constant across all countries, despite considerable 
variation in the settings in which children study French, and if there 
are good a priori reasons for considering the relation as causal, then 
the case for a causal connection is greatly strengthened. In this 
event the study will indicate which steps aimed at improving the study 
of French are likely to be rewarded. (Of course underlying certain 
variables - e.g. sex of student - are factors which cannot be altered. 

These variables provide information which will be of interest to educa- 
tionalists and psychologists.) On the other hand, if regression varies 
from one country to the next, and in particular if the coefficient asso- 
ciated with some predictor is zero in some countries, the connection 
between the factor underlying this variable and attainment in French 
is likely to be a complex one. This in turn would mean that measures 
to improve French teaching directed towards this factor may not have 
the desired result, or may only produce it when other factors interact 
with the changes in some unknown manner. In short, it would be wise to 
delay such measure until further research had determined under what circum- 
stances the factor had a bearing on instruction in French. 

II. Hypotheses 

The following are examples of hypotheses which might be tested: 

1« (i) Variation in achievement profile will reflect variation 

in teaching methods and objectives: children taught French by audio- 

visual methods will reach a higher level of achievement in speaking 
and listening skills, children taught French by grammar-translation 
methods will reach a higher level of achievement in reading and writing 

skills • 

(ii) This variation will not persist into the pre-university grade 
(if length of exposure to French is held constant.) 

2. Level of Achievement in French will be related to length of 
exposure to French rather than to chronological age or grade level. 

3. Attitudinal factors (in both home, and school environment) will 
affect level of achievement in French. 

4. Socio-economic factors (parental education, socio-economic skills, 
etc.) will be related to child's level of achievement in French. 

5. Girls will reach a higher level of achievement in French than 

will boys: this difference will be accentuated in single-sex schools. 

6. Size and composition of French class will be related to achieve- 
ment in French. 

7. Selection procedures will be related to level of achievement 

in French: children in streamed schools will achieve a higher mean score 

than will those in comprehensive systems; the latter will produce a higher 
total "yield" but a lower average score. 
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Children’s level of achievement in French will be related to the 
training, qualifications, and teaching experience. 



9. Variation in economic support for education will be related | 

to variation in level of achievement in French. 1 

1 

10. Children’s level of achievement in French will be related to ! 

parental aspirations and attitudes. j 

I 

i 

J 

11. Level of achievement in French will be related to opportunities f 

for contact with French speaking groups. I 

J 

s* 

1 

12. Level of achievement in French will be related to the ‘'ling- § 

uistic distance" between French and the mother-tongue. J 



13. Measures of verbal ability in the child's mother-tongue will 
correlate highly with measures of achievement in French at the advanced 
levels; in the early stages of learning, there will be a much lower 
correlation. 

III. Variables 

The following are a very preliminary listing of some of the variables 
likely to be used in hypotheses testing. 

A. National Variables 

1. National Educational System 

a. Types of schooling available. 

b. Percentage of each age-group in school 

c. Percentage of each age-group in the different types of school 

d. Age of beginning school - Minimum school-leaving age 

e. Expenditure on education and educational research 

f. Perceived status of teaching profession 

g. Type of administration: central/local 

h. Inspection system: type and extent 

2. Specific linguistic factors 

a. Number of native languages spoken in the country 

b. History and extent of French teaching in the country 
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c. Social factors influencing attitudes towards French: religious, 
economic, etc. 

d. Official policy towards the teaching of French: stated objectives 

e. Status of French as an occupational or examination requirement 

f. Estimate of "linguistic distance" between French and the mother 

tongue and identification of major sources of interference: reading direc- 

tion, orthography, etc. 

B. School Variables 

1. Type (primary, secondary - state/independent, etc.) 

2. Sex (boys', co-ed, girls') 

3. Size: total enrollment; size of classes; number of classes; age-range 

in each class; number of teaching staff 

4. Method of selection or recruitment 

5. Organizational pattern (streamed/ comprehensive) 

6. Location: urban/ rural 

7. Curriculum: sugjects taught; choice possibilities degree of special- 

ization; number of languages taught; stage at which languages intro- 
duced and to whom (whole ability range?) status of French in curriculum, 
French used as a medium of instruction for other subjects? 

8. Contacts with France and the French: school visits; "twinning"; club 

activities; availability of French books, films, etc. 

C. Teacher Variables 

1. Personal data 

a. Age 

b« Sex 

c. Professional training: length and type; formal qualifications 

d. Extent and type of training for French teaching; residence in 
France? Is training perceived as relevant to French teaching objectives? 

e. . Number of classes taught; size of classes; age— groups within 
classes; selection of children within classes; time allocated to French 
lessons. If teacher in primary school, class teacher or French specialist? 

If class teacher, how many hours is teacher absent from own class to teach 
French? Is class taught as a whole or in groups? 
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f. Support for French teacher: in-service training; access to 

language laboratory; contact with French-speaking groups; availability 
of French books, films, etc. 

g. Teacher’s salary scale 

h. Teacher's attitudes towards France and the French. Perceived 
objectives of teaching French 

2. Methods and Materials 

a. Methods used to teach French: direct/indirect/combination of 

methods, etc. (Possibility of developing a scale to specify teaching 
methods with reasonable accuracy.) 

b. Sequence of French teaching: listening, speaking, reading, 

writing? Investigation of specific teaching emphases 

c. Course m^ierial used: audio-visual course, textbooks, teacher's 

own materials, etc. 
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d. Availability and use of funds and equipment (language laboratory, 
tape recorders, film-strip projectors, etc.) 

D. Pupil Variables 

1. Age 

2. Sex 

3. Socio-economic background: parental occupation; level of parental 

education; size of family, etc. 
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Linguistic background: use of French in the home; contact with 

French outside school program; number of languages spoken by the 
child; number of languages learnt by child in school 



5. 



Age at which child introduced to French; number of hours' exposure 
to French teaching (split, if possible, into the amounts of time 
devoted to each of the four basic skills) 



6. 



Motivation of child for learning French: perceived objectives 

for study of French; level of aspiration; attitudes towards French 
language and people; attitudes towards child's own culture 



7. 



Child' perception of parental attitudes towards education in general 
and towards learning French in particular (investigation of parental 
attitudes desirable, if feasible parents' interest in and knowledge 
of French; attitudes towards child's learning French; perceived 
objectives for study of French) 
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8. Child's verbal ability in mother tongue 

9. Child's disabilities of hearing, vision, or speech 

10. Child's homework; total amount and amount of French homework 
IV. Definition of the Target Populations 

The design of the study requires that representative samples of stu- 
dents of French be drawn from each of the participating countries in such 
a way that there will be a maximum degree of comparability among the samples, 
within and among countries. This "maximum'' degree of comparability may in 
fact be quite limited because of the varying policies concerning French 
instruction in the different countries or in different sectors of the educa- 
tional systems of those countries. Nevertheless it will be important and 
desirable to attain whatever degree of comparability is possible. 

The experience of the IEA study of mathematics suggests that the chief 
way in which some degree of comparability can be attained is to sample with 
respect to the age or grade of the student. But it is not practicable or 
even desirable to sample at all ages. Only two or three age— level cohorts 
can be sampled. • 4 

A major constraint upon the sampling which is particularly relevant 
in the case of an "optional" subject like French is the amount of time the 
student has been exposed to the language. It is pointless to try to measure 
achievement in French for a student who has not taken any instruction in the 
subject. A useful measurement for the purposes of this study can be taken 
only after the student has had approximately two years, or about 160-180 
contact hours of instruction. (For all students included in the study, infor- 
mation will be gathered on the amount of instruction, in years and/or contact 
hours.) The samples would also include, wherever possible, students who have 
continued the study of: French for more than two years. For example, a sample 
of students at a given age-level, say 16 years, would include some who would 
have studied French for as long as 5 or 6 years. (It would not include, how- 
ever, students who started 5 or 6 years earlier, studied for 2 years, and 
then dropped it.) 

To enable the International Committee to determine appropriate age-level 
samples, therefore, country— by— country statistics are needed on the numbers 
of students of each age-level who have studied French for at least two years 
and who are still enrolled in French. If it is impossible or difficult to 
collect statistics by age-level, statistics according to grade-level are 
acceptable., even if only in terms of percentage. 

An example of the sort of table that is desired from each participating 
country is given on the next page. 
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Age (or 
grade level) 


Total number of 
pupils in the age- 
level (or grade level) 
whether studying 
French or not 


(Estimated) Number (or Percentage) 
of students who enrolled in French 
and who have studied for: 

2 yrs 3 yrs 4 yrs 5 yrs 6 yrs . . . 
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I It may be useful to set forth a tentative set of target populations based 

| on the Committee's general impressions of instruction programs in the several 

| participating countries. Three sampling points seem to be suggested: 

I 

| Age 10 (approximate grade level: end of grade IV) uecause of the Committee's 

\ interest in the teaching of French in the elementary school (as a result of 

; the conclusion of the May, 1966 conference at the Unesco Institute for Educa- 

i tion) , a sampling at this age level would yield information concerning how 

much can be achieved at this age level and what factors are associated with it. 
It might not be possible to collect samples of this age level in all partici— 
l pating countries, however. 



Age 14 (approximate grade level: end of grade VIII) The participating countries 

have programs which introduce French as a foreign language anywhere from grade 
IV to grade VII. By the end of grade VIII there would be in each country sizable 
numbers of children who have studied French for at least two years. 







Age 17-19 (pre-university level) In each of the participating countries, there 
would be substantial numbers of students who would have studied French as a 
foreign language for at least two years and are still enrolled in it. Of those 
enrolled, some would have studied for many more than two years. 
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APPENDIX G-2 



French Test Rationale and Description of Pre-Tests 



General Considerations 

The test descriptions presented here reflect in large part the tenta- 
tive framework for test development previously described in the minutes 
of the I.E.A. International Sub-Committee meeting held at Educational 
Testing Service, Princeton, New Jersey, from July 31-August 4, 1967* 

A meeting of the U.S.A. National Committee for French was held in 
Princeton some weeks later (September 18-19 » 19&7) » this committee 
reviewed the proposed specifications in considerable detail* On the 
basis of these meetings, and taking account of various other comments 
and su g gestions sent in by other national committees, Mrs* Clare 
Burs tall and Dr. John Clark assembled several hundred draft test items 
for presentation at the meeting of the full International Committee held 
in London from October 9-13, 1967* During this meeting, considerable 
discussion was raised concerning the general theory and orientation of 
the tests, and certain modifications were suggested for some parts of 

the tests. A detailed description of the various points raised is pre- 

sented in the minutes of the International Committee meeting, and will 
not be repeated here* In addition to the general discussxons, the Com- 
mittee also edited the actual test items and checked the stylistic and 

technical accuracy of these materials* 

The test descriptions and rationale presented below are intended to show 
the composition and measurement functions of the pre-test materials as 
they have been conceived and assembled to date* These materials are to 
some extent a synthesis of two basic approaches to foreign language 
testing: on the one hand are included various item types which attempt 

to measure the student *s general command of the language as reflected in 
his ability to comprehend spoken or written materials presenting various 
"real-life 1 * linguistic situations, as, for example, conversations between 
two or more people, radio broadcasts or loudspeaker announcements, and 
representative reading texts of paragraph length or longer. On the 
other hand, an attempt has also been made to provide a number of items 
which lend themselves to close analysis in terms of the linguistic 
features which they incorporate* Items of this type can serve a diag- 
nostic function in that they allow the tester to determine specific 
language behaviour which students have acquired or have failed to ac- 
quire in their course of study. Items testing the accurate pronuncia- 
tion of certain French phonemes or the written mastery of certain verb 
forms would exemplify this second approach. It seems fair to say that 
both approaches to evaluating language competence are valid in their own 
right and serve a complementary function in the overall testing endeavour. 
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A number of potentially useful testing ideas which are not reflected in 
the draft tests have been presented, both orally during the various 
meetings and through correspondence with the national centers. We wish 
to acknowledge receipt and consideration of these suggestions; however, 
in view of the restraints imposed on the testing program, both in terms 
of the total testing time available and also in consideration of the 
problems involved in the controlled administration of certain types of 
test materials, it has been necessary to adopt a rather economical ap- 
proach in selecting item types and content for inclusion in the tests. 

A number of interesting but somewhat peripheral aspects of language 
competence (for example, the ability to deal with dictee materials, or 
to read poetry aloud) have necessarily been omitted in favour of closer 
attention to the more basic language skills. 

Description and Rationale for Individual Tests and Item Tjrpes 

POPULATION^ONE TESTS 

The tests for Population One (10-year-old level) will be administered 
only in England, Scotland, and the U.S.A. At this level, most students 
will not have had sufficient training in writing to justify the inclu- 
sion of a Writing test. Thus, tests for Population One will be limited 
to Listening Comprehension, Reading, and Speaking (N.B. - For Population 
One, as well as for each of the other populations, the Speaking tests 
will be administered only to a sub-sample of students in view of the 
administrative problems involved in tests of this type). 

Listening ComprehensionJTest: The basic testing technique will be the 

tape-recorded~presentat!on”of single sentences, spoken by a native French 
speaker using a ’’standard’ 1 style of pronunciation (i.e , free from marked 
regional characteristics). Speed of delivery will be sufficient to in- 
sure natural liaison and intonation. After listening to each sentence, 
the student will choose from a set of four pictures printed in his test 
booklet the one which correctly depicts the spoken sentence. 







* 



It should be noted that this technique eliminates any element of reading | 

comprehension; this is particularly important of the 10-year-old level j 

since, depending on the training program followed, many students will 1 

have had little exposure to reading by this time. \ 

Depending on the ways in which the answer options are planned and manipu- 
lated with respect to the spoken material, the four-picture format can 
be used to test both vocabulary knowledge and syntactical control, par- j 

ticularly for number, gender, and subject-object distinctions. f 
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The Listening Comprehension test will require about 30 minutes, includ- 
ing administration time, and will contain 50-60 items. At this level, 
students will mark their answers directly in the test booklet rather 
than use a separate IBM answer sheet. A short test break is anticipated 
about half way through the test. 

Reading Comprehension! The Reading test at this level will also make 
use”o?”a”^-pic ture”f ormat , except that the stimulus sentences will now 
be printed in the test booklet in boldface print. In the final section 
of the test, a few brief passages of continuous prose will be presented, 
and each passage will be followed by two or more printed multiple-choice 
questions based on the passage. This section is intended primarily for 
the most advanced students in the sample, and its inclusion will help 
to avoid the possibility of a "ceiling effect" for the test. 

The lexicon used in both the Listening and Reading tests has been drawn 
from Francais Fondamental. ler degre . An attempt has been made through- 
out to use only those vocabulary items which could be presumed common to 
the various teaching programs used at this level. 

The Reading test will require about 20-25 minutes overall and will con- 
tain approximately 50 items. 

Speaking! The Population One Speaking test will consist of three sec- 
tions7~Fronunciation, Structural Control, and Fluency. In the Pronun- 
ciation section, the student will listen to and initiate a number of 
brief but complete and meaningful sentences (e.g. , "La neige tombe"). 

The sentences are selected to exemplify selected details of pronuncia- 
tion (principally, the accurate production of vowel and consonant pho- 
nemes) and intonation (interrogative, declarative and exclamatory pat- 
terns). Although it might be possible to score the production of each 
sound or intonation pattern along an extensive continuum of quality, it 
is considered more practical and reliable to use a simple three-point 
scale, or perhaps, simply to score the items a s right or wrong. 

i 

The Structural Control section presents a number of single pictures de- 
picting a -simple scene (e.g. , a child drawing a picture). The student 
is asked to reply orally simple questions relating to each picture (e.g. 
Que fait la petite fille?). The pictures and questions are designed so 
that only one specific answer or a very restricted range of answers 
would be elicited. The emphasis of this section is thus not on the 
student's ability to "talk" in French from a fluency point of view but 
rather to produce an appropriate and grammatically correct response to 
a well-defined stimulus. 
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The final section of the test - Fluency - presents single but fairly 
complex pictures which the student is asked to describe in some detail. 
Both overall fluency and grammatical accuracy will be scored according 
to 3- or 4-point scales defined in verbal terms. 

The Speaking test will be administered on an individual basis using two 
tape recorders, one for presentation of the instructions and aural stimuli 
and one to record the student’s responses. Centrally located scoring, 
possibly at Education Testing Service, is envisioned for the Speaking 
tests at all levels. 

The timing for the Population 1 Speaking test will be 10-15 minutes, 
including ins true t ions • 

At the present time, draft tests have been prepared in Listening, Speak- 
ing, Reading, and Writing at the Population 2-3 level (both of these 
populations will take the same tests) and at the Population 4 Advanced 
("specialist") level. General survey tests at the Population 4 level 
have not yet been assembled, since test development has proceeded on the 
assumption that the survey test will be made up, for the most part, of 
selected items from the Population 2-3 and Population 4 advanced tests. 

Within a given skill area, the same general item types have been used 
for both the Population 2-3 and Population 4 Advanced tests, although 
the general level of item difficulty is of course greater in the latter 
case. The format descriptions which follow may be considered as apply- 
ing to the tests at both levels. 

Listening^Comgrehension: The first section (LC/PICT) of the test pre- 

sents”4-picture"’panels" together with an aural stimulus of sentence- 
length. Basic vocabulary is tested, as well as certain important syn- 
tactical elements (see, for example, items 1, 4, 107, 108 ). These pic- 
ture items are intended to provide a pleasant and interesting introduc- 
tion to the Listening test as a whole. 

The second section (LC/SENT) consists of a number of simple statements 
or questions such as would be encountered in everyday conversational 
situations. In the student’s test book are printed four short state- 
ments, one of which represents the most normal or appropriate reply to 
the spoken sentence. 

It is realized that this technique involves a reading comprehension 
element, which would not be the case if the answer options were them- 
selves spoken (as had originally been planned in the July 31- August 4 
sub-committee report). However, it became apparent upon discussion 
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that the use of spoken answer options was problematical for several 
reasons. First, since the spoken options would necessarily have to 
follow one another on the tape, students might be able inadvertently 
or deliberately to signal the correct answer either by marking their 
answer sheets immediately after the correct answer is given or by 
making some other sort of gesture. Second, the spoken-option technique 
was considered somewhat uneconomical of testing time, in that a total 
of 3 sentences would have to be read aloud in order to present a single 
item. For these reasons, the committee considered that it would be more 
useful to present the options visually, with the proviso that the answer 
options themselves be couched in such simple terms that the student for 
whom the Listening test per se would be appropriate would not have dif- 
ficulty in reading the answer options. Persons reviewing the listening 
items are urged to keep this aspect in mind, and to point out any items 
which they would consider to present an unfair reading load. 

The third section of the Listening test (LC/CONVER) is similar to the 
second except that now the student listens to a short conversation be- 
tween two people and checks the option which correctly describes what 
is happening in or is implied by the conversation. Depending both on 
the linguistic complexity of the conversations themselves and the par- 
ticular questions asked, the level of difficulty of these items can vary 
from extremely simple and straightforward to highly challenging. (As a 
general rule, the items at the beginning of any given section are de- 
liberately quite easy, and gradually increase in difficulty throughout 
the section, although occasional easier items are interspersed to keep 
the weaker students from becoming discouraged.) 

The fourth section of the test (LC/PASSAGE) present short radio or loud- 
speaker announcements, reclames, and so forth, followed by a spoken 
question on the overall situation or on some detail of the situation. 
Again, the difficulty level of the items is varied within the section. 

The fifth and final section (LC/LONG CONVER) consists of longer conver- 
sations or dramatic scenes which continue for one or two minutes, and 
which the student is expected to listen to as a whole. Following this 
material, the student is asked a number of questions based on the pas- 
sage. The questions are intended to test basic comprehension of the 
passage, to the extent that it would be retained by an interested na- 
tive speaker, as contrasted to the recollection of minute details which 
even the native speaker would probably not remember. 

With few exceptions, the items sppearing in all sections of the listen- 
ing test should be considered as testing general comprehension at various 
levels of linguistic complexity, rather than the control of certain 
specified elements of phonology, syntax, or lexicon. 
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Reading JTests: The Reading tests, as presently established, consist of 

two” sections" vocabulary-in-context (RC/VOGAB) and reading passages 
(RC/TEXTS). Although it would appear that the comprehension of written 
texts, as tested in the second section, is closest to the tasks facing 
the student in real-life situations, there is a slight drawback in the 
use of items of this type (or at least, in their exclusive use) in that 
it is difficult to sample within any reasonable amount of testing time 
all the levels of difficulty and extent of lexicon which it would be 
useful to cover* As a consequence, a preliminary section consisting of 
a fairly large number of discrete reading items, generally one or two 
sentences in length, has been provided# These so-called "vocabulary- 
in-context" items, present a short passage or sentence from which a 
single word or group of words has been omitted# It is the student* s 
task to select from among four printed alternatives the word or phrase 
which best fits into the stimulus sentence# This item type lends itself 
to the testing of a number of important reading elements, including the 
vocabulary of nouns (of various degrees of difficulty), adjectives, ad- 
verbs, verb forms, and idiomatic constructions# In some items, the 
point of difficulty lies in the options themselves, whereas the stimu- 
lus phrase itself is quite straightforward; in others, the four options 
presented are easy to understand, but in order to choose the correct 
option, a close and accurate understanding of the stimulus phrase is 
required# Rather than detailing here the various possibilities of this 
item type, it is suggested that the test reviewer look through these 
items at leisure to determine the different aspects of comprehension 
involved# 

As previously stated, the second section of the test presents longer 
reading passages of various degrees of complexity and in varying written 
styles# Two or more printed questions, each with four answer options, 
are based on each passage. Since in the Reading test the student is 
able to refer back to the passage (as contrasted to the situation for 
listening comprehension passages), it is here possible to ask detailed 
questions about the content of the passage, as well as to ask more 
general questions on the theme or mood of the passage# 

The Reading tests are anticipated to each require about 30 minutes' 
testing time, including instructions. As they are here presented, 
there is a slight surplus of Reading test items; the final forms will 
probably incorporate somewhat fewer items in each of the two sections# 
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WritingJTests : Unlike the Listening and Reading tests, which are for 

the"most"part addressed to problems of general comprehension, the Writ- 
ing tests offer the possibility of testing the student's command of 
specific details of syntax, morphology, and lexicon. It should also 
be pointed out that the Writing tests measure the student's language 
proficiency from a production point of view (as do the Speaking tests), 
whereas the Listening and Reading tests constitute measures of receptive 
skills. 
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In specifying item types for the Writing test, it was considered essen- 
tial to allow the student to construct vthat is, actually write out) 
his responses, rather than to choose from among multiple options. On 
the other hand, it was necessary to insure - for the sake of scoring 
reliability - that student would respond uniformly and predictably to 
the materials presented (assuming, of course, that they had a correct 
knowledge of the points being tested). Thus, the first and major sec- 
tion of the Writing tests (Wfy/GRAM) at both levels was designed to 
present partially completed sentences, each of which omits a single 
word (noun, verb, adverb, adjective, conjunction, preposition) bearing 
on the particular syntactical or morphological problem being presented. 
Although the student does not in this part of the test produce complete 
sentences, he is nonetheless required to produce that portion of the 
sentence which is considered to represent the essential problem. Since 
the sentences are relatively short and require only one-word fill-ins, 
a fairly large number of such items can be presented in a reasonably 
short time span. 

The second section of the Writing tests (WI^PEON) is designed to test 
the student's ability to handle various collocations of subject and 
object pronouns, both for proper form and correct location in the sen- 
tence. The general technique is to present a model sentence containing 
a number of nouns (e.g., "Est-co que Jacques prete sa bicyclette a son 
frere?") and to ask the student to rewrite the sentence substituting 
the correct pronouns ("Oui, il la lui prete"). The difficulty level 
of the items is determined by the number of nouns to be replaced, as 
well as by the form of the sentence (affirmative or negative) /and the 
tense of the main verb (present or passe compose). 
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The third section of the test (Mfy/VBS, MODIF) evaluates the student's 
ability to produce correct verb forms (regular and irregular) for 
various persons, and to form correctly various descriptive adjectives, 
both regular and irregular. The testing technique is to present a model 
sentence, and then to change one or two words (e.g., from "homme" to 
"femmes"); this change in turn requires changes in verb forms and/or 
adjectives in order to produce a correct sentence. 



The final section of the Writing test (WiR/DIR COMP) gives the student 
the opportunity to produce a "free composition," albeit within certain 
necessary limitations. The general testing procedure is to give a 
series of stimulus-words which suggest a short story. The student 
writes out his story, using the stimulus words presented, and in their 
order of appearance. This section of the test is scored for inclusion 
of all the necessary elements, and for the grammatical and stylistic 
accuracy of the sentences produced. Although this last section is the 
least "objective" in format, prior experience with this item type in- 
dicates that it can be evaluated by properly trained scorers with a 
satisfactorily high degree of reliability. 
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Speaking Tests: The Speaking tesits at the higher levels correspond in 

general outline to the Population One Speaking test, except that cer- 
tain other item types are added* Following the Pronunciation section 
appears an Oral Heading section in which the student reads a short para- 
graph aloud* Scoring is based on the accurate pronunciation of certain 
phonemes and on proper intonation, but particular attention is also 
given to required liaisons* across-sound modifications (such as the 
denasalisation of un bon ami ) , and other such features of the "Chaine 
parlee." 

The Structural Control section is basically similar to that for Popu- 
lation One, except that the pictures and questions are designed to 
elicit more complex responses* 

The Fluency section' is broken into two sub-sections: the first sub- 

section presents a panel of three pictures depicting a simple event. 

The student is a&ked to "tell the story that the pictures tell," and 
it is anticipated that the responses will all be in the present tense* 

In the second sub-section, a single picture is shown and the student 
is asked to describe what probably happened prior to the time of the 
picture, what is happening at the present moment, and what the probable 
outcome will be. This technique allows the student somewhat more re- 
sponse latitude, and at the same time checks his command of past and 
future tenses* 

An effort will be made to hold the higher level tests within a 15 min- 
utes total time limit, including the time required for the additional 
sections* 
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CONCEPTUAL FRAMEWORK 
FOR 



CIVIC EDUCATION 



As a result of the recent IEA International Civic Education 
Committee meeting held at the end of April, 1967, this document has been 
prepared in order to feedback to National Civic Education Committees t 
present state of the work and also to make further requests to National 

Committees . 



4. 



This document is in six sections as follows . 



A" Working Definition of the Content and other Domains of Civic 
Education - (This is a result of an analysis of the National 
Committee reports submitted to the International Committee 



The International Master Grid for the Content and Abilities of 
Civic Education: A Graphic Approach to Blueprints for Testing 
the Target Populations 



a) The Master Grid 

b) Topics by Target Populations: 

Population I 
Populations II and III 
Population IV 



3. Sample Questions 



Progress on Work Concerning Attitudes, Values, and 
Background Questionnaires 



a) Summary of Values 

b) The Measurement of Attitudes and Values in 
Civic Education 



5. Hypotheses 



6 . 



Requests to National Centers 

(As will be seen from Section 6, one of the main requests 
is for items to be written. In this connection special 
attention should be paid to Sections 2 and 3) . 
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We have documented the fact that civic instruction is given in 
varying degrees under various titles in all countries. It may be 
called history, geography, or social studies in England, Australia or 
Germany, for instance, and it may be given as a separate subject, such as 
in the United States, Iran, Sweden, Italy, or France for example. In 
Sweden, of course, "Civics as a school subject is dealt with in two 
ways: as a separate subject , and as a form of work in general , regardless 
of [the] subject being taught." (Marklund, Instruction in Civics in 
Swedish Schools , Page 5). In other words, civic education may permeate 
other subjects although, nominally, the subject being taught may be 
"economic," "social studies," or the like. Similarly, the committee 
fully recognizes the variety of civic instruction within countries, 
depending upon school type (private, public, comprehensive vocational, 
technical, academic, rural, urban, etc.). Allowances for this variance 
will be made in the content specifications, the testing instruments, the 
background information, and the interpretation of results. 

The committee does not mean to imply that civic education is 
synonymous with the "social studies" broadly defined. Instead, within 
the social studies, there are aspects of geography, history, economics, 
and other studies which are closely akin to studies carried on in 
civics, citizenship, government, problems of democracy, international 
relations, current events, and other clearly identifiable civic education 
subject. The aspects of the social studies included in this study are 
those concerned with human political interrelationships on various 
governmental levels in organized societies . 

By civic education, then, we mean that instruction which influences 
the political socialization of the child. That there are formal and 
informal influences on this education (family, school, peer group, mass 
media, etc.,) goes without saying. We shall attempt to sort out such 
sources and to weigh their influence. At the moment, however, the 
committee is concerned with identifying those phases of formal education 
which help develop knowledge, understanding, attitudes, and behaviors 
related to governmental affairs at all levels (local to worldwide) . 

To repeat, those other institutions (economic, religious, family, 
etc. ,) which influence the socio-political life of the student are 
certainly part of one’s governmental education, are certainly important, 
and must be assessed. However, the committee has chosen to identify 
those school subjects at the various age levels having political 
sofcence content and to make an over-all list of the major topics 
(e.g., democracy, constitutions, community, etc.,) covered in many 
or all nations . 



1. A Working Definition of the Content and Other Domains of Civic Education 

As planned in Bulletin 1 (December, 1966), the Civic Education 
Committee has examined the twelve nations’ unique concepts of civic 
education. As we said there "After content analysis we intend to 
revise our present concept of civic education." The rationale 
statement and model concerning power, decision making and deference 
components, affective, cognitive, and behavioral domains and relation- 
ships from the level of selfhood to the world were present as a general 
a priori scheme of analysis for comparative educational research. 

This approach is well recognized and is familiar to students of 
comparative government. 

The committee has been most pleased with the many useful responses 
from the National Centers regarding their answers to our first four 
questions in Bulletin 1. These replies have aided us in redefining our 
concept of civic education. The two guiding questions for this redefi- 
nition were: 1) "Is education for citizenship in the curricula of all 

countries or not?" and 2) "Does a sub ject^called "civics" exist in all 
of the countries engaged in this inquiry?" 

We have conducted a content analysis of the documents received 
(including supplementary material on France and Israel) . This ja 
posteriori , or inductive approach, has had an effect on our 
hypothesized schema or picture of international civic education. 
Nevertheless, we had anticipated many of these results owing to 
the international composition of the committee and members’ general 
knowledge of comparative educational systems. 

From the start, the civic education committee recognized the 
difficulty of defining the content, cognitive, affective and behavioral 
domains of the field. Thus, it was in November, 1966 that the committee ^ 
first changed its name from the "civics" committee to the "civic education 
committee, before beginning work. This was necessary because the mean- 
ing of the word "civics" differs among and even within, countries. The 
committee conceived of its task as being the study of political education 
or instruction in govenment or political science in the twelve participat- 
ing nations - whatever title may be utilized in the country itself 
(citizenship, civics, political education, social studies, political or 
constitutional history, occupational orientation, etc.) 



1 Macridis , Roy C., The Study of Comparative Government . New York: 

Random House, 1955 (pp. 34-62). 

2 See Civics and European Education. Strasbourg: Council of Europe, 

1963 (pp.150). 
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For purposes of this study, the definition of civic education 
for a country may partially be determined at each age leve^ y t e 
appearance (or indication) in a content-knowledge category or a na i na 
statement or goal which signified that that objective is significant 
for and characteristic of, instruction at that age level. A subject 
which may be titled work orientation, for instance, may be characters ic 
of elementary school instruction in Sweden and Italy, but not in other 
countries; thus, this topic might be examined by national centers in 
those countries and no others. On the other hand, a topic like e 
United Nations may be part of every nation's civic instruction and will 
therefore be part of the final school year examination in every country, 
as well as in those earlier years where the goals, curriculum guides, 
national or other laws, and tests detail such instruction. 

The civic education committee has compiled an over-all content domain 
description (attached) resulting from our international content analysis. 
We realize that certain content topics may be mentioned in national 
goal statements, but not taught, and others may not be mentioned, and 
vet taught. The real crucible will be found after the testing itself, 
the interpretation of test results and the identification of those 
influential controlling or explanatory factors (formal or informal) 
as a result of attitude scaling and background questionnaires. 

Thus, the committee decided to identify major content areas and 
cognitive abilities (e.g. , ability to comprehend and interpret politica 
data), then to ask which aspects of power, decision making and deference 
apply to certain of the major content topics (e.g., central and national 
government), and finally to determine if this is important and appropnat 
for measurement in different countries at the various age levels. e 
recognize the need to be highly selective in sampling from the expan- . 
sive universe of material which could be tested in terms of power, deci- 
sion making, deference, content, ability, age level, and level of a s- 
traction (self to world). As a first step, therefore only the most 
representative content and abilities dimensions have been described 
for each age level and level of abstraction. Categorization of ques- 
tions into overlapping clusters dealing with power, etc., will have 
to await the actual production of test and attitude questions. n 
other words, power, decision making, and deference will both become 
central themes for the organization of the actual international civic 
education content and in terms of the questions themselves. 

2. Th e International or Master Grid for the Con tent a nd Cognitiv e 

Domains of Civ ic Education: A Graphic Appro ach to Blueprint s^ 

for Testing the Target Populations . 

After the content analysis revealed that the word civics (or 
"citizenship") was a meaningful educational term in all countries 
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(but with varying degrees of currency) it was decided to look for the 
least common denominator of political content among all the partici- 
pating countries. The committee examined various national defini- 
tions of civics, political education, or social studies and found each 
of them either too connotatively overgeneralized (even for an inter- 
national study) or too country— specif ic in their denotative meanings . 

The search for the philosopher’s stone, the perfect and all encompas- 
sing definition suitable to all national centers, was soon aborted, 
however. The present definition was labelled both sufficient and 
necessary. Instead, the committee looked to the content (both overt 
and covert) of political instruction as defined by various countries, 
and selected those elements of the various social studies curricula 
which had political relevance (political geography , government ec<j- 
nomic policy, political history , etc.) Thus it was that one master 
content-abilities grid was prepared from the country-by-country ana- 
lysis. Specific goals for age levels and for the attitude area as 
a whole (discussed below in Section 4) were also prepared. 

2a. The Master Grid 

The following grid is a general summary of the topics and abi- 
lities to be tested, as a result of our analysis of the national com- 
mittee reports. 

No cell weighting has been entered into the master grid but in 
general it is expected that at the lower age levels the amount of 
application and synthesis test items will be at a minimum, and percep- 
tion and conception will be emphasized. By the upper age levels the 
number of application and synthesis items will comprise a larger pro- 
portion of the examinations. This plan fits in, it is assumed with 
both the psychological ability levels and the content level of abs- 
traction for the age groups • This is to say that peer group , family , 
school, local community, etc., will be stressed at the lower age levels 
and national and international areas at the higher age levels. It is 
hoped that a detailed weighting for each cell can be arrived at over 
the summer of 1967 as a result of detailed analyses of national reports. 
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A) 



B) 



Nature of Citizenship 

1) Concepts of sovereignty, polity, stage, 
political communities, nation, loyalty, 
democracy, etc. 

2) Rights, duties, obligations, and respon- 
sibilities (e.g. minority rights and ma- 
jority rule) 

3) Individualism, freedom, authority, rule of 
law, cooperation, leadership, participa- 
tion, etc. 

Political Processes and Institutions 

1) Local, departmental, state (or inter- 
mediary levels) , national and compara- 
tive government (including political- 
historical development) 

2) Political processes-public opinion, poli- 
tical parties, pressure groups, elections, 
etc. 

3) The legislative branch - law making, 
constitutions, etc. 

4) The executive branch, including the 
bureaucracy (civil service) 

5) The judicial branch and courts. 

6) Governments and social services, e.g. 
insurance, working conditions and unem- 
ployment, social security, welfare, educa- 
tion, medical care, health, roads, housing, 
personal security, etc. 

7) Foreign policy, national defense; and in- 
ternational relations and organizations 
(e.g. balance of power, collective securi- 
ty, UNO, the emerging nations, etc.) 
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Economic Processes and Institutions 

1) Production, distribution, and 
consumption. 

2) Industrial organizations, labor 
unions, and agriculture. 

3) Money and banking, the market, 
inflation, and public finance 
(taxation) . 

4) Insurance, demography, natural 
resources, the world of work, and 
occupations . 

5) Government and the economy, economic 
trends, standard of living, etc. 

Social Processes and Institutions 

1) The political socialization process 
(including family, school, community, 
peer groups, etc.) 

2) Communications, mass media, trans- 
portation, and advertising. 

3) Recreation and leisure time, group 
action, traffic, crime, welfare and 
service institutions. 





Topics 

The topics as given in the master grid cover all age ranges and the 
list is inclusive. As will be seen from the three lists of topics for the 
different target populations given in the next section only some parts of 
the grid are used, particularly for the 10 year old population, but these 
parts are often expanded. 



Note: A detailed international frequency analysis will be calculated 

in the summer of 1967 with specific illustrations of each content divi- 
sion drawn from the national reports which specifically identify (in 
statement form) the content, level of abstraction, etc., for each age 
level. 
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Recognizing the great diversity in civic education content in the 
various nations, the committee decided to keep the recall, selection, and 
recognition level at a minimum. Certain facets of current events, in- 
ternational relations and foreign policy , and international organiza- 
tions can safely be measured on the higher recall, selection, and recog- 
nition levels. However, the following categories of cognitive-abilities 
dimensions (which actually encompass or subsume the recall, selection, 
and recognition of factual information) were agreed to: 



| I. Perception : The ability to comprehend and interpret political 

| data and information concerning power, decision making, and deference and 

| to gather, receive and extract from such political data. The student: 

| comprehends and understands political information; has an insight into 

; the meaning of data that goes beyond mere recall; illustrates skills of 

| translation of political symbols (e.g. political party symbols); can sum- 

[ marize and interpret a publication; demonstrates the ability to extra- 

| polate trends . 
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II • Conception : The ability to analyse and evaluate political situa- 

tions, problems, and information concerning power, decision making, and 
deference . The student: determines and applies the criteria to choose 

among meaningful alternatives, to make a political judgement about excel- 
lence in political writing to estimate progress toward social goals, to 
weigh different plans of action, or to determine the best criteria for 
distinguishing among democratic and autocratic political systems. 

III. Application : The ability to apply the results of political 

analysis and evaluation to political problems concerning power, decision 
making, and deference . The student: demonstrates the skill to apply 

theories or abstractions discussed in a study to a specific or concrete 
political event (or events) and proves his ability to handle statistical 
information and other social science tools of analysis useful for the 
study of public problems. 

IV . Synthesis : The ability to synthesize political information 

concerning power, decision making, and deference. The student: de- 

monstrates his ability to distinguish fact from opinion; to compare 
political subjects, systems, and generalizations in order to produce an 
original hypotheses or plan of action; to generalize with insight; to 
derive a set of abstract relationships; and to test hypotheses or modify 
them on the basis of new informations. 

The selection of this list of major cognitive areas was done with 
the full knowledge that much of the extant testing materials in various 
nations is rendered useless because of its factual recall or recognition 
orientation. Most of the materials which are appropriate to measure these 
areas of perception, conception, application, and synthesis, for instance, 



0 

ERJC 






H-8 












would require open ended or brief essay questions they may not be measured 
in this study. However, it is expected that the examples provided below 
and those which are now available in various national examinations (in 
Sweden, Australia, and the United States, for instance) will provide some 
of the material needed for the pretests to be constructed in late August, 
1967. 



2b . Topic by Target Populations* 

In this section a figure is placed in parentheses after each major 
head. This indicates the rough weighting to be attached to each major 
head and therefore gives National Civic Education Committees an idea of 
the proportional number of questions to be produced for each sub-section 
(4= major emphasis; 3* considerable emphasis; 2= some emphasis; and 1= 
little emphasis) , 

(i) Population 1 (10 year olds) 

A. Nature of Citizenship (4) 

1. Duties and Responsibilities: 

a) Respect for others and their property (including property) 

b) Politeness (kindness) 

c) Helping others (younger, old and handicapped persons) 

d) Law observance, propriety, rules, conventions 

e) Participation 

2. Rights and freedoms (to receive a-d above) 

a) Existence and survival 

b) Worth of the individual 

c) Member of society 

d) Right to persuade, speak, reason, (a-d above), and to 
participate 

B. Political Processes and Institutions (1) 

1. National symbols and songs 

2 . Local government 

3. Central government 

4. International organizations (UNO, Red Cross) 

5. Peace and conflict 

C. Economic Affairs (2) 

1. Education, work, and recreation 

2. Consumership and saving 

3. Taxes 

4. Property 

5. Advertising (mass media) 



* For definition of Target Populations see Bulletin 2 (May 1967) pp.2-3 
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D. 



Political-Social Processes and Institutions (3) 



1. Family life 

2. School life 

3 . Peer groups 

4. Communications (mass media) and transportation 

5. Housing, traffic planning, armed forces, hospitals, public 
security, welfare, and service institutions 

(ii) Populations 2 and 3 

A. Nature of Citizenship (2) 

1. Concepts of sovereignty, polity, state, political communities, 
nation, loyalty, democracy, etc. 

2. Rights, duties, obligations, and responsibilities (minority 
rights and majority rule) 

3. Individualism freedom, authority, rule of law, cooperation, 
leadership, participation, etc. 

B. Political Processes and Institutions (4) 



I 

-i 



1. Local, departmental, state (or intermediary levels) national 
and comparative government (including political-historical 
development) 

2. Political processes-public opinion, political parties, pressure 
groups , elections 

3. The legislative branch-law making, constitutions, etc. 

4. The executive branch, including the bureaucracy (civil service) 

5. The judicial branch and courts 

6. Governments and social services, e.g., insurance, working con- 
ditions and unemployment, social security, welfare, education, 
medical care, health, roads, housing, personal security, etc. 

7. Foreign policy, national defense, and international relations 
and organizations (e.g., balance of power, collective security, 
UNO and the emerging nations.) 

Economic Processes and Institutions (3) 

1. Production, distribution, and consumption 

2. Industrial organizations, labor unions and agriculture 

3. Money and banking, the market, inflation and public finance 
(taxation) 

4. Demography, natural resources, the world of work and occupa- 
tional choice 

5. Government and the economy,, trends, standard of living, etc. 




D # Social Processes and Institutions (2) 

1. The political socialization process (including family, school, 
community, peer group, etc.) 

2. Communication, mass media, and advertising . e tc 

3. Recreation and leisure time, group action, traffic, crime, e . 

(iii) Population 4 

A. Nature of Citizenship (3) 

1. Basic political concepts of sovereignty, polity, political commu- 
nities, nation, loyalty, etc. ...... 

2. Legal rights, freedoms, duties, obligations, and responsibi 

ties, e.g., minority rights and majority rule 

3. Individualism, freedom, authority, rule of law, cooperation, 

leadership, etc. 

B. Political Processes and Institutions (3) 

1 Local, departmental, state (or intermediary levels) national 
and comparative government (including political-historical 
development, separation, division, and interrelationships, 

2. Political^ processes and policy formulation - public opinion, 
political parties, pressure groups, and elections 

3 . Central government 

a) The legislative branch - law making constitutions, etc. 

b) The executive branch including the bureaucracy (civil service) 

c) The judicial branch and courts 

4. Governments and social services, e.g., insurance, w ° rk ^f “ n ^"_ 
tions and unemployment, social security, welfare, educati , 

cal care, health, roads, housing, personal security, etc. 

5. Foreign policy, national defense, and international. relatio 
and organizations (e.g., balance of power, peace and conflict, 
collective security, UNO, underdeveloped nations, etc.) 

C. Economic Processes and Institutions (2) 

1. Production, distribution, and consumption 

2. Industrial organizations, labor unions, and agriculture 

3. Money and banking, the market, inflation and public finan 

4. Insurance! demography, natural resources, the world of work, 

5. Government*" and the economy, economic trends, standard of living, 
etc . 
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D. Social Processes and Institutions (1) 

1. The political socialization process (including family, school, 
community, peer group, etc.) 

2. Communications, transportation, mass media, and advertising 

3. Recreation and leisure time, group action, crime, traffic, wel- 
fare, service organizations, etc. 

* * * 

3. Sample Questions 

The following tentative, illustrative items have been suggested for 
Population 1 (10 year olds) . It is hoped that they will be useful to mem- 
bers of national committees when they come to write items. On the left 
hand side of each item is a code indicating the content (A, B, C, or D 
in the topics by Target Population's chart) and the ability (I, II, III, 
or IV - see master grid) . Under each item in parentheses is an indica- 
tion of the general categories (power, decision making, and deference) 
identified for organizational sub-scores. It should be noted that these 
items are only a few of those reviewed by the international committee, 
but it is hoped that many better items will be written. (it is hoped 
to send out illustrative items for a) Populations 2 and 3 and b) Popu- 
lation 4 in the very near future.) 

1. In your country the government makes all children attend school 
for so many years. Why does the government do this? 

D2, (A) Because fathers and mothers who work cannot take care 

of their children. 

(B) Because it is dangerous for children to play on the streets 
all day. 

* (C) Because children must be taught to be educated members of 

your society. 

(D) Because your teachers need jobs. 

(E) Because your teachers and the school principal (or headmaster) 
have decided so. 

(Power, Decision Making) 

2. Stimulus material: Line drawing of a policeman apprehending a 

rich man and a vagrant who have picked flowers in a public park. 

What is happening in this picture? 

Aid, (A) Two people are hurt and another man is helping them. 

I (B) Three people are playing a game in the public park. 

(C) Three people are picking flowers in a public park. 

* (D) A policeman is taking two men to jail for picking flowers in the 

public park. 

(E) A policeman is hurting two people who have done nothing wrong. 
(Power) 
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3. 

Aid, 



* 



4. 



B2a 

3, 
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* 



5. 



B3, 

IV 
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6 . 



Cl, 
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7. 




What does this picture tell your about the law? 

(A) Only policemen can pick flowers in the public park. 

(B) Only policemen obey the law, 

(C) All taxpayers should be able to pick flowers in the public parks. 

(D) The law is for all people, rich and poor alike. 

(E) Everyone should learn to obey the law because a policeman may 
be nearby. 

(Power) 

Stimulus material: Line drawing of a courtroom trial with bench, 

robed judge or judges, etc. 

This is a picture of a (an) 

(A) school 

(B) office 

(C) library 

(D) trial (or courtroom) 

(E) hospital 
(Power) 

Which of the following things does your government make, and that 
no one else in your country is allowed to make? 

I. Postage stamps and money 
II. Passports and drivers licenses 
III. Bank check and bonds 
IV. Flags and guns 

(A) I and II only 

(B) II and III only 

(C) III and IV only 

(D) I, II, III, and IV 
(Power) 

Stimulus material: Line drawings of each of the following occupa- 

tional groups. Which of the following people work for the govern- 
ment? Circle (a) if the person works for the government, (b) if he 
does not, and (c) if you do not know whether or not he works for the 
government . 



a 


*b 


c 


A milkman 


*a 


b 


c 


A public school teacher 


*a 


b 


c 


A soldier 


*a 


b 


c 


A congressman, M.P., etc 


*a 


b 


c 


A policeman 


*a 


b 


c 


A judge 


*a 


b 


c 


A postman 



(Power 

Stimulus material: Line drawing of Italia Con Corona Turrita, Brit- 

tania, Uncle Sam, Adler, etc., or picture of national paper currency 
with an appropriate national symbol. 

is dressed like he (she) is because this 
(A) figure stands for a symbol of your nation 
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(B) is the way people dressed long ago 

(C) shows that your country belongs to the United Nations 

(D) shows that the money was printed by the government 

(E) is patriotic 
(Deference) 

8. We must put stamps on our mail in order to 

C2, (A) avoid paying a fine 

II (B) help stamp collectors in other countries 

* (C) pay the government for delivering mail 

(D) show that we are good citizens 

(E) prove we know the mail regulations 
(Power) 

9. Peter usually walked to school, but today he was riding with his 
father because he had stayed in bed too long. He told his father, 
"Hurry, father, you are driving too slowly. I will be late for 
school and that is against the rules." "Sorry, son," said his^ 
father. "Thirty-five kilometers (miles) per hour is the speed 
limit, whether you are late or not." 

Aid, What is the best reason why Peter's father should obey the speed 

III limit? 

(A) The police may see him 

(B) The law sets the speed limit 

* (C) The danger of an accident is less 

(D) The people who speed are punished 

(E) The car engine will be harmed if it goes too fast. 

(Decision Making) 

10. Who made the law about the speed limit? 

Aid, (A) The drivers 

I (B) The police 

(C) The judges 

* (D) The government 

(E) The automobile companies 
(Power) 

11. What must every citizen do even before he becomes a voter? 

* (A) Pay taxes if the government requires it 

C3, (B) Go to religious services regularly 

I (C) Join a political club or party 

(D) Go on to a college or university for further education 

(E) Memorize the Constitution and the national anthem 
(Power) 

12. Which of the following can best be determined after a school class 
votes on it? 

A2d , (A) Deciding the best way to make a sick classmate well again 
III (B) Deciding whether a small chicken which someone has brought into 
class is a male or female 



II 
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D2, 

Ala, 



14. 



Aid, 

IV 



(C) Deciding what the price of bread in the school lunchroom should 

(D) Deciding which student in the class has the best examination 

(E) Deciding who shall be the class representative or officer 
(Decision Making) 

Which of these students shows most clearly that he has not been 

a good citizen in the classroom? 

(A) One who has not done his homework 

(B) One who has stolen something from another classmate 

(C) One who has stumbled and broken a pane of S lass 

(D) One who has been responsible for his team losing in a class game 

(E) One who has misspelled some easy words 

(Deference) 

The children in a class had made their own rules. 

They agreed to do the following things. 

(A) To put up their hands when they wanted to speak 

(B) To keep themselves and their desks tidy 

(C) To keep to one side on the stairs and in the corridors 

(D) To stop others from throwing stones, sticks, or anything 

that might hurt someone . ^ . 

(E) To speak politely and in a courteous and friendly manner 

others 

(F) To do their homework regularly , oai1 

(G) To eat in a mannerly way and to keep the lunc roo 

What are the letters of those rules which will protect pupils from 
accidents? C & D 



What are the letters of those rules which will make you a good 
citizen of the school if you follow them? 

All 



15. 



D2, 

Aid, 

IV 



(Decision Making) 

Rules are made by different people for different reasons. Some rules 
are made by governments for all people. Which rules below are made by 
the government? Circle (a) if they are made by the government or 
(b) if they are not, or (c) if you do not know who makes the rules. 



*a 

a 

a 



a 

*a 



b 


c 


Traffic regulations 


*b 


c 


Regulations for playing sports 


*b 


c 


Lunchroom regulations 


*b 


c 


Rules for table manners 


b 


c 


Voting regulations 



(Power-Decision Making) 
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16. Why does Johnny have to study in school? 

Dl, (A) Because otherwise his parents will not take him on vacation 

I * (B) Because it is his job to learn many things which will be use- 
ful to him 

(C) Because he is the best student in the class 

(D) Because the government makes him go to school 

(E) Because the other children study 
(Decision making) 




17. Suppose that Johnny finds a wallet on the ground, what should he 
do? 

(A) Give it to a poor man 

(B) Keep it for himself 

(C) Give it to a policeman 

(D) Divide the money among his friends 

(E) Take part of it for himself and give the rest 
back to the owner 

(Deference) 
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4 . Progress on Work Concerning Attitudes, Values and Back - 
ground Questionnaires . 

During the Rome meeting a separate subcommittee was es- 
tablished to work on the areas of attitudes and behaviors in 
civic education. This subcommittee will be working in con- 
junction with the international committee. Attitude scales, 
background questionnaires, and similar instruments will be pro- 
duced conjointly. The subcommittee on attitudes (which will 
receive additional help from the IEA technical committee and 
from other experts in the field) was composed of Dr. A. N. 
Oppenheim of the London School of Economics and Mr. T. N. Post- 
lethwaite, the IEA coordinator. 

4a . Summary of Values 

The following summary is the result of a survey of the 
national papers in order to detail the general values which 
appear time and again. (A more complete list with list of 
values is being compiled). 

Democratic Values Stressed In International Civic Education 
Documents . 

1. Tolerance of diversity and egalitarianism. The idea 
that all citizens are of equal value as human beings and that 
their differences are a source of interest and strength to 
the country. 

2. Acceptance of interdependence and involvement: na- 
tional, international (UN), and historical. The idea that no 
man is an island, and that people are connected with one an- 
other in groups, as members of states both at the international 
level and with their past. 

3. Respect for the balance between the liberty of the 
individual, freedom of the press, assembly, worship, express- 
ion, etc., versus the rights of others. The ideals of equal- 
ity before the law, civic rights, nondiscrimination, and self- 
restraint . 



4. Attitudes about the use of and limitations on 
state power, accountability of elected representatives, 
voting, and political behavior. 



5. Attitudes toward certain functions of the state 
in (1) providing services, social justice, welfare, secur- 
ity, planning, etc., (2) regulation of competing interests, 
e.g., antimonopoly legislation, trade unions, and consumer 
protection . 

6. Acceptance of the ideal of progress and social 
change through informed participation versus the feeling 

of powerlessness; social concern, and a rational and criti- 
cal approach to social problems; work for the common good; 
incorruptibility; informed public opinion. 



7. Belief in the pacific regulation of conflict; 
peace; nonviolence, group decision making process.es; major- 
ity versus minority rule. 



8. Belief in the need for participation in school, 
family and group decisions. 

Other Values, or Those Less Emphasized or Only Emphasized 
in One or a Few Countries . 

9. Preparation for life, familiarity with the func- 
tioning of modern society; job orientation, leisure, and 
vocational opportunities, and the acquisition of new know- 
ledge independently. 

10. Love and loyalty to country; solidarity, reasoned 
patriotism . 

11. Moral principles, e.g., truthfulness, integrity, 
responsibility efficiency on the job, preserving one’s 

| health; thrift, saving, and enterprise. 



ues . 



12 . 

13. 

14. 

15. 

16. 



Cultural and aesthetic values. 

Fear of God and Christian charity, religious val- 

Unification of Europe, internationalism. 

Problems of leisure. 

Deference to authority. 



Of the values in the second section only numbers 9 
and 10 were frequently mentioned. 
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After having summarized the values mentioned in na- 
tional committee reports, Dr. Oppenheim prepared a docu- 
ment concerning the measurement of attitudes and values in 
Civic Education. This document is reproduced below. It 
suggests a special pilot study to construct some of the in- 
struments. It is hoped to begin this pilot study in the 
near future. 

4b . The Measurement of Attitudes and Values in Civic Edu - 
cat ion . 



by 

A. N. Oppenheim 

Attitudes as Dependent and as Independent Variables 

1. For the sake of completeness it should be pointed 
out at the beginning that there will be a number of attitu- 
dinal measures included among the predictors of all subject 
achievement. In the area of Civic Education, such predic- 
tors might include, for instance, some indication of paren- 
tal attitudes, or the type of discipline favored by the 
teacher. The attitudes are to be measured in this way, i. 
e., as independent variables. A list of these will provide 
the subject of a separate document. What follows is pri- 
marily concerned with attitudes as dependent variables; in 
other words, these are the attitudes and values which we 
hope (or expect) to find developing in children, partly as 
a result of school teaching processes and partly as a re- 
sult of various aspects of socialization generally. 

2. We recognize that there are close links in this 
domain between perception, knowledge, values, and behavior- 
al tendencies, with influences going backwards and for- 
wards - such that an absorption of knowledge is partly se- 
lective in line with pre-existing attitudes, but also that 
further knowledge may influence attitudes and values. If 
we seem to separate attitudes from knowledge, this is pure- 
ly done for purposes of convenience and should not imply 

a misleading dichotomy. 

The Contents of the Country Documents . 

3. At the outsat there was no indication as to whether 
or not there might be a common core of values and attitudes 
which are present or implied in the teaching programs of 
all or most countries involved in the research. It was 
therefore necessary to conduct a fairly detailed content 



analysis of the country documents in order to ascertain 
the various values that were being emphasized, at dif- 
ferent age levels. The results of this analysis (see 
previous document) tended to show that all countries con- 
cerned emphasized a broad area of democratic values. In 
4a these values have been specified under 8 headings, 
which are fairly arbitrary in their divisions. We have 
tried to indicate the degree to which each of these in 
turn is emphasized in each country, according to the docu- 
ments supplied to us. Of course, we recognize that dif- 
ferent countries have interpreted their task differently, 
and that the documents do not necessarily always reflect 
accurately what is classroom practice. Moreover, in the 
field of attitude transmission much is done by implication 
rather than by direct tuition. Much depends on the way 
in which factual material is taught, as well as on the 
actual content. 

4. In addition to this broad area of 8 major points 
covering democratic values, we found that a number of 
other attitudes were mentioned (by one or more countries) 
as having some emphasis in the curriculum. (See list of 
other values 9-16.) For instance, fear of God and Chris- 
tian uharity were mentioned in one country, problems of 
leisure were mentioned in another, and so on. Possibly 
some of these additional values could be encompassed in 
the broad framework of democratic values, while others 
might have to remain separate or might otherwise become 
the subjects of country-specific attitude measures. 

Approaches to Measurement 

5. It now becomes incumbent upon us to stand back 

from the material and to ask ourselves: "How can these 

detailed statements derived from the curricula be linked 
with existing knowledge about democratic attitudes?" It 
would, of course, be possible to go ahead and produce 10 
or more attitude measures, each covering a sub-area of 
democratic values. But it might be possible, and perhaps 
more valuable, to develop one or two broad, underlying 
measures. The advantages of so doing would be threefold: 

i) it would reduce testing time considerably 

ii) it would link the measures in this project 

to existing psychological knowledge, and make 
a contribution to it. 
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iii) 



the overt contents of such underlying measures 
would be somewhat less Apolitical in content > 
thus making it easier to secure permission to 
have them administered to school populations 



6. Over and above these three advantages there is 
another, which is specifically of relevance to the aims of 
IEA; namely that it may be possible to develop these broad, 
underlying measures in 'a form that will make intercountry 
comparisons possible and valid. If this were so, then, 
leaving aside for the moment problems of translation, etc. , 
we could hope to end up with country measures or country 
profiles that would be comparable. The results of the in- 
quiry might hope, then, to show two things: 



i) 



the extent to which pupils at different age levels 
in each country had progressed along measures of 
democratization 



ii) 



the extent to which each country, in turn, had 
achieved, exceeded, or fallen short of its own 
stated aims in this area. 



7. It is worth pointing out that we expect to deter- 
mine if the school is or is not one of the major influential 
factors having impact on the development of these values 
in the child. We may find that open socializing processes 
will, in this domain, come more to the fore than elsewhere. 
In this study we expect to find that the influence of the 
school will vary, by degrees, from country to country and 
from age group to age group even within a country. Ulti- 
mately it would be helpful to governments and other read- 
ers of our reports to know to what extent democratic values 
were developing in the children, through whatever means, 
and which are irrespective of the school curricula. 



8. The area of democratic-authoritarian values is one 
that has long been the subject of attention is social psy- 
chology and other social sciences. Many early studies were 
brought into sharp focus in the 1940's by the work of Adorno 
and others on the authoritarian personality. The F-scale, 
which was a result of this massive work, covers the type 
of underlying variables that were referred to above. It 
includes, for instance, the tendency to use stereotypes and 
black and white thinking, tendencies to pr o j ec t ivi ty , em- 
phasis on power and strong leadership, fatalism and so fort 
The broad stream of research which followed this study has 
shown that scales of this kind are closely related to the 
more detailed democratic values in civics (which are men- 
tioned in 4a) for adults. There have been studies of 
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authoritarianism and rigidity in children - mostly attempt- 
ing to link the development of such attitudes to various 
types of regimes in the home but no generally accepted and 
validated F-scale for children exists at the moment. 



9. In this context mention should be made of later 
work such as that of Rokeach on the open and closed mind, 
and the work of Eysenck on basic social attitudes. Neither 
of these research workers has produced scales directly 
appiicabie to children, however, Eysenck's work has two 
possibls o. d v a n t a g 6 s to our studyi 



i) It presents a broad constellation of political 

and social attitudes, including authoritarianism, 
within a two-dimensional framework of: (a) radi- 

calism/conservatism, and (b) tough mindedness/ 
tender mindedness. Within this framework high 
scorers on the F-scale find themselves located 
in the tough-minded, conservative quadrant. 

ii) The work has been repeated a number of times in 
various European countries, and the same factor 
structure has emerged; this means that while de- 
tails of items and translations may vary, the 
underlying dimensions are roughly similar, and 
scoring can take place on a basis which is com- 
parable across countries. 



Proposed Pilot Study 



iO. It is not possible to ascertain the nature of 
such underlying factors in children without conducting a 

Su £ stantial pilot study. Mounting such a study would 
P y be a separate exercise to be conducted by the 

tec ni ca l committee. It should not be assumed that attitude 
scaies are necessarily the best way of approaching this pro- 
blem, other types of items should be tried, including pro- 
jectrve techniques (if they can be made scorable) . The de- 
taiied pianning of the pilot study would have to be con- 
sidered, but a rough plan might suggest the following steps: 



i) the construction of an item pool or set 

of measures directly derived from the points 
in 4a above* 

ii) additions to the pool derived from the liter- 
ature on the use of F-scales with children, 
measures of rigidity, stereotyped thinking, and 
basic social attitudes such as those used by 
Rokeach and Eysenck, and possibly others 



I 
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iii) 



translation and administration of this 
pool to groups of children at different age 
levels in 3 or 4 selected and contrasting 
countries 



iv) 



a factor analysis or component analysis which 
would enable us, hopefully, to identify a 
very small number of basic underlying variables, 
common to the different countries and age groups, 
and scor able by factor loadings 



v) preparation of the final scales 



11. It is also worth mentioning that the conduct of 
this experiment is a major exercise in itself. It would, 
however, have all the advantages stated above and, in addi- 
tion, make an important contribution to soc io-psychological 
literature and knowledge. 



Other Values 



12. Decisions will have to be made about measurement 
of the values mentioned under 9-16 in 4a. Some of these 
may already be subsumed under other democratic values; 
others may have to be measured separately, and still others 
may fall by the wayside. 



13. We also have to bear in mind some broad mental 
attitudes which have been suggested to us by others, such 
as : 



i) 



tendencies to stereotyped thinking; (this 
is easily subsumed under authoritarianism 
and rigidity) 



ii) 



measures of self-esteem, as a student and as 
a human being (like other attitude measures, 
this in particular could be both a dependent 
and an independent variable) 



iii) 



measures of need— achievement (This refers to 
the research of David McClelland who has 
related broad measures of achievement-motiva- 
tion in children and in adults to the degree 
of achievement-oriented content of mass media 
to a country’s social and economic achievements, 
e t c . ) 
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5 . Hypotheses 

National Committees will have seen from Bulletin 2 
the formulation of composite hypotheses being undertaken 
by IEA. The variables to be entered into composite hypo- 
theses can be gleaned from the consideration of a series 
of shorter hypotheses. In November, 1966, Sixteen Mark- 
lund of Stockholm, Sweden wrote a short memo "Concerning 
Hypotheses" which will be of interest to National Committees. 
This is reproduced below. 



Concerning Hypotheses 
by 

Sixten Marklund 



A. The outcomes of civic education can be described 
and evaluated in many different ways and according to 
various dimensions: It is of prime importance at the be- 

ginning of such a study to obtain a rough indication of 
what civic education is or at least of what it is thought 
to be. The IEA civic education committee therefore needs 
information from participating countries about how they de - 
f ine civic education. Three factors are of main importance: 
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1. Objectives of civic education 

2. Content of civic education 

3. Subject organization in relation to civic 
education 



B. As far as pupil achievement is concerned, there 
seems to be three main areas of interest: knowledge, atti- 
tudes and interest in the subject. Continuing the numbering 
from Section A they can be broadly sub-divided as follows: 





4. 


Knowledge 


of 


facts, total mean 


f 


5. 


Knowledge 


o f 


facts, profile of subscales 


i 


6. 


Knowledge 

subscales 


of 


facts, dispersion in scales and 


i 


7 . 


Attitudes 


(feelings, etc.) in human relations: 

f amily 


* 


8. 


II II 




peers 




9. 


II * II 




school and teachers 


i 


10. 


II II 




certain groups (minorities, etc.) 


% 

?■ 


11. 


II II 




police and other public officials 




12. 


II II 




international affairs 


i 


13. 


Pupil interest in civic education. 



l 
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For all of these domains evaluation instruments must 
be constructed with which quantitative data can be collec 
ted and subsequent analyses carried out. The test and 
scale items must concern important concepts and features 
of human and political relations which are common to all 
countries (e.g. power, deference, freedom and decision 

making) . 
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C. The domains of variables thus arrived at (4 
13 above) can be treated as dependent variables. The 
independent variables should, of course, be detailed des- 
criptions (in measurable terms) of the points 1, 2, and 

3 mentioned in Section A as well as variables such as 

the following: 

14. Amount of civic education teaching at 
different age levels 

15. Percentage of pupils of an age group taught 
civic education at different levels 

16. Categories of teachers, categories and infor- 
mation on their pre-service and in-service 
training 

17. Methods of teaching in civic education 

18. Tests and examinations in civic education 

19. Types of civic education in different school 
types, e.g., comprehensive, selective, academic, 

etc . 

20. Differences in civic education for boys and girls. 



D. This list is far from complete and the variables 
can, of course, be made operational in a much more detaile 
and precise form. Further pupil and teacher back||£und 
variables can be added such as those used m the IEA mathe- 
matics study. 

E. More information in each of these areas must be 
collected. This data collection will constitute a ne- 
cessary first phase in the civic education study. The 
generating and developing of hypotheses is strongly depen 
dent on what this first phase yields. 
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The hypotheses will in general refer to a great num- 
ber of int e-corr elat ions between these variables. Differ- 
ences in measurements will be found within countries as 
well as between. 

F. The folloitfing are, therefore, only a few examples 
of hypotheses which may be tested in the study. They 
illustrate suggested types of hypotheses rather than final 
hypotheses : 

1. Pupil knowledge of governmental affairs is 
higher when civic education is given as a 
separate subject than when it is an overall 
aspect of a number of subjects. 

2. The pupils' interest in participating in 
social activities is positively related to 
their interest in governmental affairs. 

3. Pupils' participation iti extra-school acti- 
vities (clubs, games, etc.) is related to 
their perception of authority relationships 
within the family. 

4. The more a curriculum in civic education 
stresses a functional approach (as opposed to 
a structural approach) the greater will be 
the variability of the knowledge-profile 
pattern. 

5. The variability of knowledge of international 
relations is positively related to the degree 
a curriculum in civic education stresses a 
structural approach. 

6. The pupils' interest in the subject of civic 
education is higher when instruction is given 
in short, concentrated courses than when in- 
struction is spread out over a whole year 

(or more) . 

7. Pupil difference between age 13 (+1) and age 
17 (+1) in attitudes towards participation in 
school activities is positively related to the 
type of school organization which promotes 
such activities. 
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There is a negative correlation between the 
pupils’ knowledge of international organizations 
and their thinking in stereotyped terms. 



9. Etc. 



6 . Request to National Centers 



1. We would like to remind you of the information 
requested for Phase B (see Bulletin 1, pages 83-84, items 
5 - 20) . (Some National Centers have already answered 
some of the questions posed for Phase B in their Phase A 
reports. Other centers have stated that they have not the 
resources to conduct the analysis needed to answer certain 
questions. If such information is not readily available, 
the expert or professional judgments of the national civic 
education committees will suffice for answering the spe- 
cific query . ) 



2. (a) We would like to have further information on 

what political content is taught at specific age levels to 
specific portions of the school populations. 



(b) At the same time we would appreciate receiving 
selected outlines of the political material (from text- 
book tables of contents, curriculum guides, governmental 
syllabi, etc.) taught to each age group. We do not want 



any more than a summary of geography, sociology, economics, 
history, etc., material which is taught unless this ma- 
terial has a political content, e.g., political geography, 
public opinion polling, international political history, 
public finance, etc. 



(If you require any elucidation of what is requested 
in 2 a and b please write to Dr. Russell Farnen, Educa- 
tional Testing Service, Test Development Division, Prince' 
ton, N.J. 08540, U.S.A. 



3. (a) Many items are required for try-out, to this 

end we would like as many examples of currently used ex- 
aminations with certain items categorized as to content, 
ability, age level, etc. in terms of the grid supplied in 
Section 2 above. 



The international committee has (and intends to con- 1 - 
tinue to do so) reworked examinations and questions sub- 
mitted by national centers. However, the types of ques- 
tions sought for this international survey are sufficiently 
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unique as to be scarce in the literature. National cen- 
ters are encouraged to use appropriate questions which 
appear in the other national reports they have received 
as prototypes for such questions. Centers are also wel- 
come to adapt those sample questions which they find are 
appropriate to their nations or to modify them into a 
format suitable for use in their own nations. As these 
questions are written, the international character of 
these tests much also be kept in mind. Those national 
questions which may be used in several countries are the 
most valuable. The use of stimulus materials (maps, 
charts, graphs, cartoons, line drawings, quotations, etc.) 
is to be encouraged in this way the examinees will have 
some measure of common verbal or nonverbal material on 
which to base their responses. 

4. We would be interested in receiving the national 
centers comments on the materials presented to them thus 
far (grids, sample items, rationale, etc.) The same in- 
vitation applies to the material on attitudes presented 
below . 



5. All materials produced should be sent to the IEA 
coordinator in Hamburg before July 30, 1967. The next 
meeting of the International Civic Education Committee will 
be held at the end of August, 1967. 

We thank you for your help. 



Russell Farnen 
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APPENDIX H-l 



Civic Education Content Specifications - 
Cognitive Domain 8/29/67 (Hamburg, W. Germany) 
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I II III 

A. Fundamentals and Nature of Citizenship - 1 

Citizenship concepts (definitions). 

1. Sovereignty, polity, state, govern- 
ment, political communities, nation, 
loyalty, patriotism, nationalism, 
legitimacy, authoritarianism, etc. 

2. Rights, duties, obligations, responsi- 
bilities, and power (e.g. minority 
rights and majority rule), etc. 

3. Individualism, freedom, authority, 
rule of law, cooperation, leader- 
ship, participation, democracy, 
amity, trust, equality of opportuni- 
ty, etc. 

B. Political Processes and Institutions 

1. Constitutions 

2. Historical (Social-Political- 
Economic) development, including 
political theory, separation of 
powers, etc. 

3. Local, departmental, state, province 
(or intermediary levels) as compared 
with central government, etc. 

4. The national legislative branch, law 
making, etc. 

5. The national executive branch (includ- 
ing the bureaucracy, civil service), 
the administrative branch, and the 
Cabinet, etc. 

6. The national judicial branch and 
courts . 

7. Political processes and institutions, 
public opinion, political parties, 
pressure groups, elections, political 
coalitions and decision making, etc. 

8. Foreign policy,, national defense and 
foreign affairs, etc. 

9. International relations and organiza- 
tions, cross-cultural, nonwestern, com- 
parative politics and governments (e.g. 
unitary and federal systems, parliamen- 
tary and presidential systems, etc.) 
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APPENDIX H-I (cont'd) 



10. Governments and social services e.g. 

insurance, working conditions and unem- 
ployment, social security, welfare, ed- 
ucation, medical care, health, roads 
housing, personal security, etc. 

Economic Processes and Institutions 

1. Government and the economy, economic 
trends, standard of living, foreign 
aid and trade, technology, etc. 

2. Industrial organizations, labor unions, 
and agriculture. 

3. Money and banking, the market, insur- 
ance, inflation, and public finance, 
(taxation) . 

4. Demography, natural resources, the world 
of work, and occupations. 

Social Processes and Institutions 

1. Group interactions between the individual 
peer group, family, school and community, 
interdependence, social interchange, and 
"socio-political" man. 

2. Communications, mass media, transporta- 
tion, advertising, etc. 



traffic, crime, welfare, and service ins- 
titutions . 
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Age Levels 
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Methods of the social studies disciplines, 
and Interdisciplinary Approaches i.e. How do 
the various social sciences study all the 
above, e.g. the economist, the historian, 
the political scientist, etc. 

1. Methods of the Social Sciences 

Critical thinking and problem solving 

Current events 

Note: Factors such as current events, the problem solving method, 

and the like are meant to cut across all of the above areas (A-E) 
but are categorized as F and G for test assembly purposes. 
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APPENDIX H-2 



Illustrative Items for Civic Education 



Population I: 

1. According to the rules of democratic politics, victory in an election 
consists of securing the: 

a. support of businessmen 

b. most qualified candidates 
•l-c. largest number of votes 

d. greatest financial contributions 

e. support of the minority party 

This question applying to national political processes and insti« 
tutions (II) asks the ten year old student to identify the democratic prin- 
ciple of majority rule. It is expected to be abstract for the ten year 
olds because of the selectivity required rather than the language level. 

2. Stimulus material: Pictures of five flags the UNO flag, the inter- 
national Red Cross or Red Crescent flag, the Olympic games flag, and 
the Boy Scouts and Qirl Scouts flags labeled flags 1, 2, 3, 4, and 3. 
Which of these flags is the United Nations flag? 

+a. Flag 1 

b. Flag 2 

c. Flag 3 

d. Flag 4 

e. Flag 5 

The recognition of the UNO flag by the 10 year old would show the 
beginning level of familiarity with international organizations (III). It 
is expected to be a relatively simple task for all countries involved 
including West Germany which is not a UNO member. 

3. Why must we put stamps on our mail? 

a. to avoid paying a fine 

b. to help stamp collectors in other nations 
+c. to pay the government for delivering mail 

d. to show the government that we are good citizens 

e. to show everyone that we know how to mail letters 

This question will be somewhat complex for 10 year olds. It is de- 
signed to test the child's understanding of why we must pay for governmental 
services (IV). There should be few difficulties with translation of this 
item since it is a fairly universal practice across the seven nations in- 
volved in the civic education assessment. 



4. What is the most important duty of the police in a community? 

a. to punish criminals 

b. to hand over criminals to police centers 

c. to put thieves in prison 

+d. to help people and protect their lives and properties 

e. to arrest people and take them to court 

This question for 10 year olds tests social processes and institu- 
tions (V) and is complex for the 10 year old* The question asks for the 
most important duty of the police and each of the other distractors is 
either plausible or an actual practice of the police. The translation of 
the key words police and community should pose no problem in any country. 



Population II 



1* Which of these would most probably be the best citizen in a democracy? 

Someone who: 

a. has good health and much money 

b. v/ins many athletic contests 

4c. reads about current events and votes in elections 

d. is born into a good family and has many friends in important places 

e. has had much schooling 

This simple question for 14 year olds tests citizenship fundamentals 
(I) in terms of informal political participation. It is expected to be a 
simple question across countries since it deals with a statement of a de- 
mocratic ideal which is commonly taught in the participating nations. 

2. Which of the following correctly describes your country's Constitution? 

4a. Federal and unwritten only 
4b. Unitary and unwritten only 
4c. Unitary and written only 

4d. Federal, written, remaining powers lie with federal government 
4e. Federal, written, remaining powers lie with state/provincial 
government 

This is an example of an abstract question for 14 year olds dealing with 
the national constitution and the distribution of governmental powers. The 
key will vary by nation, e.g., Iran, Italy and the UK, will be b/ Sweden and 
Finland, c / West Germany, d/ and the United States, e. It is categorized as 
national political processes and institutions (II). 
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3. Stimulus material: cartoons showing charter members and emerging 
nations on a seesaw with the U.N. building as the fulcrum. The e- 
merging nations are weighing down their end of the seesaw. 

The cartoonist is saying that: 

a. underdeveloped countries need United Nations help 
+b. there are more members who have joined the United Nations than 
there are charter members 

c. the emerging nations in the United Nations represent a larger 
total population than charter members 

d. United Nations charter members must respect the military power 
of the emerging nations 

e. underdeveloped countries do not take the United Nations seriously 
and they are only playing a game with the charter members. 

This will be an abstract item for 14 year old students in all 
countries. It tests international topics (III) and requires the inter- 
pretation of graphic material as well as fundamental knowledge of the U.N. 



Population IV (General) 



1. Citizens of a democracy are free to choose only four of the following. 
Which one are thez NOT free to choose? 

a. the religion they will follow 

b. the job they will pursue 
+c. the laws they will obey 

d. the newspapers they will read 

e. the groups they will join 

This complex question for the preuniversity year (general) school 
population asks students to select the fundamental citizenship obligation 
(I) of law observance from a list of alternatives where there is freedom 
of choice. 

2. Which of the following statements best support the argument that pres- 
sure (private political action) groups are legitimate instruments in 
the democratic political process? 

a. they generally advocate what the majority of the people desire. 
+b. they exercise the rights of petition and assembly 

c. their internal organization and mode of operation are democratic 

d. their concern is to promote the national interest 

e. they are in closer contact with the general public than are 
government officials. 
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This complex question, categorized as national political processes 
and institutions (11), associates the activities of private interest or 
pressure groups with the democratic political process. The student is 
asked to apply the democratic rights of petition and assembly to such groups 
and to accord them legitimacy. 

3. Only four of the following statements accurately describe democratic 
politics. Which one does NOT? 

a. politics is a process in which most of us take part at one time 
or another 

b. politics deal with vital issues of our time 

c. politics deal with the economic health of nations 

d. politics deal with settling conflicts among people, groups, and 
nations over the realization of goals 

+e. politics mainly deal with petty and sordid things such as how can 
propaganda be used to further general welfare programs 

This abstract question regarding national political processes and 
Institutions (II) asks the student to differentiate among four typical 
political activities and one which conflicts with usual political practices 
in democratic political systems, i.e., petty, sordid, and propagandizing 
activities. 

4. The goals of the European Common Market (E.E.C.) are mainly being 
achieved by: 

a. exchanging armed forces in member nations when needed 

b. granting more self government to the member nations 

c. stopping any products from nonmember nations from entering 
member nations 

+d. eliminating tariff barriers among the member nations 

e. granting loans to underdeveloped member nations 

This complex question on international political processes and in- 
stitutions (III) asks the student to iuentify a primary means by which the 
common market is achieving its purposes, i.e., a free trade area among the 
"Inner Six." 



Population IVS (Specialists) 

1. Which of the following principles are essential today for the operation 
of a democratic political system? 

i. rule by the wise, rich, or just 

ii. political party opposition and competition 

iii. one man or one party rule 
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IV. Universal suffrage (regardless of sex, color, religion, etc.). 

V. Periodic elections 

a. I and II only 

b. Ill and IV only 
+c. II, IV, and V only 

d. Ill, IV, and V only 

e. I, II, III, IV, and V 

In this simple question the student is asked to reject elitism, ab- 
solutism, or totalitarianism as democratic characteristics and to recognize 
the need for political competition, regular elections, and broadened suf- 
frage requirements as elements of democratic theory and practice. The ques* 
tion is categorized as national political processes and institutions (XI). 

2. Which of the following is the best electoral system for democratic 
countries? 

a. single-member district of representation 

b. proportional representation of ail parties according to voting 
strength. 

c. double-ballot, single-member district of representation 

d. suffrage -restricted to secondary school and university graduates 
+e. none of the above, because particular political systems vary 

from state to state 

This complex question asks the student to identify democratic rela- 
tivism for various political systems as well as to reject the restricted 
suffrage. He is expected to see that no one of these systems is inherently 
superior to any other and that, in fact, it is the way in which the system 
operates which is the most important criterion of political success. 

3. Which of the following were established primarily to lower customs 
duties among member countries? 

I. The World Bank 

II. General Agreement on Tariffs and Trade (GATT) 

III. Food and Agriculture Organization (FAO) 

IV. International Monetary Fund (IMF) 

V. European Economic Community (EEC) 

a. I and II 

b. II and III 
+c. II and V 

d. Ill and IV 

e. IV and V 
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This simple question dealing with international organizations (III) 
asks the student to sort out the work of certain U.N, agencies from that 
of Common Market operations on the basis of a primary purpose, that of 
reduction in customs and tariffs. The same is true for GATT, 



4. A "mixed" economic system is one in which there is: 

a. equal emphasis on manufacturing and agriculture 

b. both a national and a local tax leveled on the general public 

c. balanced government spending for civilian and military purposes 
+d. important economic decision making by both public and private 

agencies and individuals 

e. a movement from a rural to an urban society 

This complex question asks students to select the classic definition 
of a "mixed" economy from among relevant, but erroneous dis tractors. The 
mixed economy, of course, is to be contrasted with the unregulated laissez 
faire economy on the one hand and the rigidly regulated economies of e.g. 
Communist China or the U.S.S.R, on the other. It is categorized as economic 
processes and institutions (IV), 

5, Suppose that we know that since 1945 juvenile delinquency was increased 
greatly particularly in urban areas in a highly industrialized society 
which we shall call Country X. Which of the following conclusions can 
be drawn from this? 

a, peace has caused an increase in the crime rate 

b, young people in cities are basically more evil than young people 
in the country 

c, police are not as well trained today as they were formerly 

d, all young people are inherently more evil now than they were 
formerly 

+e. none of the above conclusions can be drawn 

The abstaact question for the preuniversity specialist group is de- 
signed to test the student's knowledge of certain social processes and in- 
stitutions (V), i.e., crime, its causes, the police, and the nature of con- 
temporary youth. Since none of these explanations will endure critical 
analysis, the student is expected to reject all but the last alternative. 
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APPENDIX H-3 

REPORT ON THE I.E.A. PILOT INTERVIEWS FOR CIVIC EDUCATION 

(AFFECTIVE DOMAIN) by 
Margarete Harvey 



This report is based on a series of free interviews with school 
children in England and Germany carried out in the early summer of 1967 . 
Their purpose was the identification of the main dimensions to be in- 
cluded in the affective domain of the project and, if possible, recom- 
mendations for the methods to be used. 

The procedure was as follows? during the interviews, which last- 
ed between 60—90 minutes? and were carried out on the school s premises, 
the child would be confronted with f variety of stimuli and asked to com- 
ment freely. The stimuli consisted of attitude statements used in earlier 
research, pictorial material, and introductory and/or follow-up questions 
by the interviewer. The stimulus material had been chosen to elicit res- 
ponses relevant to the "Democratic Values" as listed in Document A. , 
which was attached to CIV/RM.3. Deliberate effort was made to include 
varied approaches in order to attack the ’democratic values’ from as many 
different angles as possible 

Since there was a wealth of relevant material, roughly l/8th was 
utilized in the course of each interview, so that all the stimulus mate- 
rial was tried out at least once in each of the six schools visited. 
Moreover, some of the stimuli were only used with one age group, others 
were excluded as soon as they proved to be totally inadequate for the 
purpose outlined. 

Thg interviews were tape-recorded and then transcribed in some 
detail, listing the child's main concepts, the topics covered and sugges- 
tions for items. The following generalizations are based on the impressions 
thus gained. Since they have not been arrived at by asking identical ques- 
tions in all 58 cases, it would be misleading to give the responses in 
percentages . 

1. Sample 

The sample selection was undertaken with the view of interview- 
ing approximately 15 children in the two lower age groups (10 and 14 years) 
in both England and Germany, with a reasonable spread for sex, intelli- 
gence, and socio-economic status. 

Contact to schools was obtained through the N.F.E.R. in London 
and the UNESCO-Institute in Hamburg. Both institutions encountered great 
difficulties in getting access to schools and were in fact unable to pro- 
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vide schools for the 14 year old age group. Thus the final selection was 
more the result of expediency than of design. All the older children in- 
terviewed were German and in schools contacted privately. 

Altogether 58 children were interviewed, 38 jin the age range 10-11, 
and 20 in the age range 14-15. Of the 38 younger children 21 were British 
and 17 German. In all, six schools were visited, 4 for the 10 year olds 
and 2 for the 14 year olds. All schools were co-educational state schools. 

One of the schools for the older children was a select school, a Mittelschule, 
catering for an intelligence range in between that of the grammar schools 
(Gymnasien) and that of the secondary modern schools (Volksschulen) . 

For the younger age group socio-economic status was controlled by 
selecting one school from a thoroughly working class area and the other 
from a socially mixed area in both London and Hamburg. This procedure 
proved to be impossible for the older age group because there was no choice 
of schools and schools tend to be larger and thus draw from a less uni- 
form population anyway. In addition, the children were asked to state 
their father's occupation and the occupation they aspired for themse3.ves. 

This combined information, when broken down into the fourfold Douglas SES 
classification, gave the following distribution: 



Social 

Stratum 



Age 




I 


II 


III 


IV 


N 


10 year 
Girls 


Old: 




6 


10 


3 


19 


Boys 




1 

■ / 


8 


9 


1 


19 


14 year 
Girls 


Old: 


5 


4 


1 


10 


Boys 




1 


1 


6 


2 


10 


N 




2 


20 


29 


7 


58 



In order to overcome the teachers' tendency to present only their 
brighter students and thus gain a better idea of the spread of intelli- 
gence, the following procedure was followed: at each school the headmaster 
was asked to select one form in the appropriate age range. The formmaster 
of this form was then instructed to select the best, the worst, and two 
average students for each sex. To corroborate the teacher's estimate of a 
student's position the students were asked themselves. In all but one 
school, student and teacher estimates coincided; in this instance the head- 
master was teaching civics himself in the selected class and the formmaster 
presumably did not dare to present the dullest student. It was therefore 
arranged to see the four worst students from another form at the same school. 
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This, as well as the inclusion of two immigrant children at the 
London working class school, plus the occasional addition of 1 or 2 stu- 
dents in spare time inflated the original sample of 6 (schools) x 8 (se 
lected students) by 10 students to 58 interviews. 



It must be emphasized that the conclusions dr awn a re applicable 
to these 58 children only . 



2. Suggestions for the Main Dimensions 



As stated in the Preliminary Report, it seems advisable to split 
the broad area of Democratic Values into two big dimensions: that of demo 

cratic values proper and that of citizenship values. The reason for this 
partition are twofold: 



a) 



it is possible to learn and internalize all kinds of de- 
mocratic values, yet not be prepared to participate actively 

in the community; 



b) 



likewise it is possible to be a good citizen actively par- 
ticipating in community affairs, but to do it in a non- 
democratic manner. 



It follows from Document A that the aim of civic education in most 
countries is not just the transmission of democratic values, so that they 
are practiced at the individual level, but also their application to public 
life in short that the intended end product is an active citizen homing 
democratic values. As the’ success’ of civic education may vary in h s 
respect, yet as this very point seems crucial in an international comparison, 

it was thought necessary to split these two aspects for heur j®^ ^ P ^° S6 
To the reader of current political sociology it will seem unlikely that 
the two dimensions will vary independently, but their relationship has yet 

to be studied. 



Both democratic and citizenship values should be subdivided into 
two big areas: one a mixture of cognitive and emotive elements, the other 

consisting of emotive elements only. 



The mixed area 1 should deal with the actual kn °"^ d ?® 
zenship and democracy, where the questions actually con 

tizen" and "democracy" . The impact of civic education will most easily s 
up in this area: how the concept of democracy will gradually charge in he 

child’s eyes from a mere word with a positive ring (lOy.) to a maxim that 
has to be y upheld regardlessly (14y.) and presumably to a more orlessbalanced 
view of democracy as a system of government among a variety of others ( y*). 

how the concept of citizenship will develop from that of a merely good per- 
son with some outside interests to that of a person with rights and res 
ponsibilities . 
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Area 1 should further deal with the evaluation of democracy and 
citizenship apart from the actual knowledge about the concepts. As the emo- 
tive relationship to the concepts seems to be established before any real 
information about them is acquired, this seems to be established before 
any real information about them is acquired, this seems particularly im- 
portant. As for the democratic values questions should be asked as out- 
lined by R.B. Farnen and D.B. German under the headings ’’Government Ideas" 
and "Democratic Ideology". As for citizenship values questions could be 
limited to the desirability of becoming a good citizen. 



The emotive area 2 of both democratic and citizenship values will 



on the surface of things not deal with democracy and citizenship at all. 
It will never mention the concepts, but will include all kinds of items 
tapping the child’s readiness to act in a ’democratic’ or ’citizen-like’ 
manner. The rationale behind the division into area 1 and 2 is simple: 



a) 



the child may in time learn all the pros and cons of demo- 
cracy, he may appreciate a democratic form of government, 
yet he may not at all apply its principles of tolerance, 
equality, etc., to his own individual behaviour; 



b) 



similarly, a child may be quite sure of the attributes of 
a good citizen, yet he may not see himself as ever filling 
such a role. 



Several advantages can be gained from this division. First, one can 
see the depth of civic education in that one can compare the explicit teach- 
ing (via child’s information in area 1) with the degree of internalization 
the information has reached in area 2*. Secondly, one will get at least an 
impression of earlier socializing agents and their impact on democratic 
and citizenship values, because area 2 for the youngest age group will give 
clues about earlier socialization experiences. And thirdly, given the re- 
sults of area 2, one could form an idea about the necessity of civic educa- 
tion - assuming of course the overriding importance of the emotive aspect 
over the cognitive for the persistence of the system. 



Area 2 of the democratic values should comprehend a continuum on 
equality-inequality, for instance the child's views on and behaviour toward 
favouritism, a continuum on tolerance-prejudice (this could include a mea- 
sure of ethnocentrism, if so wished,) and a section on ’freedoms’, like 
freedom of speech, assembly and demonstration, travel, choice of job, etc. 



Area 2 of citizenship values should comprehend at least one mea- 
sure expressing the child’s feeling that his actions will matter within the 
political context, like ’political efficacy’, ’subjective competence’, and 
the ’confidence index’. It should further include a continuum on political 
apathy/participation, of obedience (acceptance, rigidity) /disobedience 
(rule-breaking, criticism) and of trust/cynicism in people and politicians. 
These subdivisions should all be in the form of continua, where the one end 
will signify a good citizen and a democrat and the other end the opposite. 
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This is not the place to speculate, but the comparisons of the ratios of de- 
mocrats to good citizens should open an interesting fi6ld for cross— national 
studies like this one. 




Apart from these two main dependent dimensions two independent varia* 
bles should be included: politisization and relation to authority. 



By politisization we mean the child’s actual behaviour toward poli- 
tical objects: his reading, viewing, discussing of political topics, his 

listening to and awareness of other people’s political views. His professed 
interest and his party allegiances belong here likewise. 



Relation to authority is a somewhat ambiguous dimension, as it 
could be treated in different ways: it could be looked upon as a dependent 

or independent variable, indeed it could be included in area 2 of citizen- 
ship values on the continuum obedience-disobedience. However, since the 
child has to establish some relationship to authority long before his school- 
ing, let alone civic education, starts, it is recommended here to treat it 
as an independent variable. Possibly it would pay to separate relation to 
non-public authority (parents, maybe teachers) and relation to public autho- 
rity (policeman, judge, Prime Minister). 



Lastly, two topics were found to be highly productive of opinions 
and speculation amongst the children: elections and laws. All kinds of 

curious misconceptions are rampant here. It would perhaps be interesting 
to include them as dimensions in the cognitive domain of civic education. 
In the affective domain they will of course be touched upon by individual 
items, but not be treated as separate dimensions. 



To summarize the recommendations for the main dimensions to be studied, 
here is a schematic drawing of them: 



Suggestions about methods 



Suggestions will be only on the type of question thought to be most 
suitable for this kind of research. 



Attitude statements shoul d be kept to a minimum at least for th e 
youngest age group. Of course , even the dullest 10 year old child 
can easily tick Yes - No, Agree - Disagree codes, but all the child- 
ren will be sorely tempted to cheat, i.e. make ticks without under^ 
standing the item. German and English children admitted that they 
would have cheated in this way, if they had been left to their 
task on their own without an interviewer asking "why" afterwards. 

In fact, some of them tried to do it in the interviewer’s presence. 
Ways to overcome this problem: 



a) 



add a category like "I do not understand the statement", or 
"I do not know word However, this creates 
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ii. 



iii. 



iv. 
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additional problems . On the one hand there is the tempt- 
ation to tick the non-committal category for many other 
reasons than lack of understanding. Secondly, the younger 
children, who doubtlessly have to tick this category rather 
frequently, will get frustrated by it. 



b) 



check the consistency of answers by comparing the scores on 
items formulated once positively, once negatively. As this 
method also helps to overcome response set, it might kill 
two birds with one stone. However, when it was used with 
the F-scale, didn’t even adults have a tendency to give in- 
consistent answers? 



c) 



again use positively and negatively formulated items^ and 
administer them as one forced-choice item. This turned out 
to be far too difficult for the children. Quite often, the 
statements were not seen as opposites and to reflect about 
two complicated sentences at the same time was quite a task 
even for the 14 year old children. 



d) 



to include some lie-detectors in such a way that a Likert- 
type item will be checked by a closed question. This can of 
course only be done for a limited number of items, but it 
should produce a measure of the truthfulness of the respon- 
ses. This would come closest to the interview situation, 
where the interviewer can follow up each answer. 



Wherever possible items should be formulated in such a way that the 
codes indicate frequency, degree, prevalence, etc. Children find 
it easy to answer questions of the "how often", "how much", "to 



what extent" type. Bronf enbrenner 's description of parental beha- 



viour is a perfect example. The Hess/Torney questions about au- 
thority figures illustrate the same point, although shorter codes 
would have been better still. 



Reading material should be kept to a minimum too. For some of the 
10 year old children reading is still a task that takes its time. 
Although this is obvious, it is a great pity, for children enjoy 
lengthy story-type questions (morale boosters) and in order to get at 
democratic behaviour, quite detailed descriptions of situations are 
necessary. So a happy medium will have to be struck on the question 
of reading material. 



Visual material is a great help for the young children, but not 
essential. Again as a morale booster an occasional picture helps, 



it is also useful if the telling of a lengthy story is thus avoid- 
ed and the child can grasp the situation at a glance as in the Rule- 
breaking Test, or where pictures of national symbols are used, which 
are familiar to the children by sight, but not by name. Yet in this 
last instance the beauty of pictorial material, the avoidance of 



H-44 


















1 






* 



1 



| 






% 



§ 



1 



rnm 














S 



i 



'$. 

I 

I 



i 



problems of translation, is lost. If questions about 
social classes are intended, a schematic drawing of 
a population pyramid with layers proved a great success. 
However, in general things can and should be left to 
the.child’s imagination. Pictures of soldiers, teachers 
or policemen are superfluous, if anything they raise 
more problems of the "our teacher does not look like 
this" type. 

v. C^o^jid_qiJ e_s t_i ori s_aj^e_b_e tj^er J^h an_o_p e n-^ml e c[ oneSj_ There 

are obviously advantages and disavantages to both. Open- 
ended questions are much better for pilot work, but they 
impose the tremendous difficulty of writing sentences 
on the child. They may have their place in the question- 
naire for the older age groups, but for the 10 year old 
children they should be excluded, if only for the time 
it takes the child to write down his answer. The closed 
question suggests answers to the child which might nev er 
have entered his mind, and it limits originality and 
variety. On the other hand, it facilitates the link-up 
of hypotheses and items, the coding and the analysis, so 
that the balance the odds are in favour of closed ques- 
tions . 

4 . Differences of Age and Intelligence 

To sum up: jdi_f fjrenjcejs ^^ijitjBljLij^ejic^ were_more _s t 

_in£ Jthan thc^se of^ jag^ wi_t h "^njteirejst i_n j>o_lijt ic s_^ £.ower^fiil iri- 

•j:erveni_n£ var^i abjLe^ 

Among the 10 year old children there were some who did 
not compare unfavourably with some of the older children, yet 
there were others who could not talk about anything further re- 
moved than their home or school. The only real and persistent 
difference between the intelligent 10 year old and the dull 14 
year old was that the older children had a larger passive voca- 
bulary to deal with political matters. Although they could not 
express their point of view any more eloquently, they could under- 
stand many more attitude statements, which will of course be a 
great help. On the other hand, this increased vocabulary can 
lead to some real confusion. For example, the older (dull) child- 
ren had been taught to differentiate be tween d emo c racy and dic- 
tatorship, and this. lesson seems to have been brought home as 
something really vital. So when talking about the state and the 
government, the following descriptions were given: State - 

that’s where everybody has a say, Government - that’s where 
one man rules. 
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On the whole the 14 year olds had of course far more 
information on topical political questions, and the teaching 
of history and civics had enlarged their knowledge of concepts. 
Yet with them the main differentiating factor was not so much 
intelligence any more, but whether they were interested in 
politics or not. The interested youngster was far more advanced; 
when asked to state his views on communism for instance, one boy 
replied: Which shall I deal with first, the Chinese, the 

Russian, or the Israeli communism?" As a result, the two most 
’ advanced f children were a boy at the select school in danger 
of having to repeat his year and a girl at the top of h'er form 
in the non-select school. Both of them could make intelligent 
responses to the Questionnaire 3.C. for students taking their 
A-levels (Primaner) . 



5. Sources of Attitudes 



Children obtain these values from many different sources 
apart fi. om their home and their school. The sources can be as 
varied, as can possibly be imagined. Visitors to the home, rela- 
tives, neighbours, mass media, recent political events and legis- 
lation, etc., etc. To cite some examples: "in a book in the 

office of my father’s boss, there was something about these men 
who tried to kill Hitler", "my auntie lives near some black peo- 
ple and you want to hear her talk about their fights", "my mother’s 
girl friend-'-s father was a NAZI, so she always had ham on her 
sandwiches", "I like listening to radio Moscow and radio London". 

It does seem from this that strange and definite sources are 
well remembered, whereas the more usual sources, from which in- 
formation flows more or less continuously, are not mentioned. 
Questions about the relative importance of home, school, friends, 
etc. , in transmitting attitudes would therefore produce rather 
misleading results. Although knowledge on other sources of atti- 
tudes is necessary in order to delineate the effect of civic edu- 
cation, which is but one source, and although it would be in- 
teresting from a therapeutic point of view, it is felt that this 
task is better not attempted. 



- A • Observations on Children’s way of Thinking 

It has to be pointed out that children seem to follow some 
peculiar patterns in their perception and understanding of the 
adult political world. Some of these patterns have influenced our 
choice of dimensions, others can be used as hypotheses, yet others 
may be useful in the selection of items and precodes. 
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i • Children tend to organize everything into good or bad 
without really knowing anything ab.out the object so 
classified . This was tried out with the concepts of 
' democracy * and 'communism'. Overwhelmingly the English 
and German 10 year olds thought democracy to be some- 
thing good and communism something bad, although hardly 
anyone of them knew anything else but the mere word. 

This suggests that emotive learning precedes cognitive 
learning. If this is so, one can hypothesize that, de- 
pending on the direction of the emotions civic education 
will either be reduced to reinforcing already existing 
notions or it will have a hard time trying to change 
these early categorizations. It also suggests a rather 
terrifyingly easy procedure to indoctrinate children. 

- The older children, while maintaining the emotional 
loading of the concepts, had something to say in justi- 
fication of their views. 

ii . Children group people in rather stereotyped big black 
and white categories. For England the categories were 
the dichotomy 'nice-horrible', which was used by the 
English as well as the two immigrant children. If peo- 
ple were described in any other terms, such as 'fighting- 
non-f igh t ing ' , or as 'giving you things or not', it could 
always be reduced to nice or horrible. In general, any 
group had nice and horrible people attributed to it. 

For Germany the stereotype was again 'nice', but this 
time with all its variations: 'not so nice, somewhat 

nice, quite nice', etc. Again, there are always nice 
and not so nice people in any group. So in a way one 
could speak of tolerance of diversity, the problem is 
that the diversity is not consciously perceived. By age 
14 this grouping into stereotyped categories had more or 
less disappeared, differences between people apart from 
nice and horrible had become apparent. 

i i i . Children work with the concept of reciprocity, or the 

belief that nothing is done on its own merit but for hard 
and .tangible reasons . For instance: "if we let immigrants 
into our country, their countries have to buy our exports; 
foreigners should be treated wej.1, because English people 
expect to be treated well abroad". This tendency seems in 
fact to increase with age, and it is by way of reciprocity 
that children first come to grips with the idea of inter- 
dependence and involvement. Starting with interdepen- 
dence between nations in terms of economics and trade, 
they proceed to interdependence between people within one 
nation: "farmers abuse businessmen, businessmen abuse 
farmers, but none can do without the other". 
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iv . Children find it hard to distinguish between the ’IS ' 
and the 1 OUGHT TO BE 1 . Whereas the majority of the 10 
year old children would never have perceived a discrep- 
ancy between the actual and the ideal, several of the 
older children made this point of their own accord, when 
asked to describe a figure of authority. For the young 
ones the way something is done is the right and proper 
way to do it. It is this that seemed to speak against 
the inclusion of a dimension like 'legitimacy'. The 
most intelligent English boy going by IQ scores, for in- 
stance, could simply not imagine an unsuccessful govern- 
ment: "it just would not be the government". -This 

phenomenon could be used in two ways. First as a tool 
by asking the children to describe an ideal and an actual 
authority figure. Secondly, as an indicator of maturity, 
if such a tool produced differences for the actual and the 
ideal . 

v . The young child is not yet spontaneously critical of 
political matters . This may not be surprising after 
what was said under point iv . and in the fact of the 
child's rather scanty information, yet want of infor- 
mation does not stop him from categorizing into good or 
bad. The little spontaneous criticism there was mostly 
came from English children. Their target was the Prime 
Minister: "him and all his laws ", he isn't nice when 

he puts up the prices". The only critical remarks from 
German children concerned the President. With increasing 
age the targets for criticism become more varied and 
the expression of it freer. This readiness to criticize 
has been included as one end of a continuum under citizen- 
ship values. 

v i , In their thinking of every aspect of governmental func - 
tions children tend to personalize and concretize. They 
either think of the incumbents of public offices as doing 
things for purely personal reasons or attribute events 
to personalities only. For instance, Johnson took over 
Kennedy's job, because he was his brother; the wall in 
Berlin was built by that chap in the goat's beard, Ulbricht. 
Institutions like the state and the government are seen 
as 'big houses' and they in turn are personalized, e.g., 
if the government made a mistake, the landlord of the 
Houses of Parliament would throw them out. In fact this 
'parliamentary landlord' figured prominently in both 
countries. -This tendency diminishes with age and it could 
therefore be used as an indicator for maturity. Where it 
persists, it occurs in a slightly more sophisticated way, 
e.g., the child would consider the possibility that his 
country was in the wrong, if he had relatives on the 
opposing side. The concr et iza t ion of institutions seems 
to disappear more slowly, as there were still some 14 
year old children, who thought of the state as an office 
in Bonn. 
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vii . The 10 year old child holds his views without any 

attempt to justify them. Whenever a child had pro- 
nounced a clear-cut opinion on an issue, he would 
be at a loss to answer the Why-question. These 
opinions were isolated ideas which the child must have 
picked up somewhere without a coherent ideology be- 
hind them, rather like the child’s party allegiance 
and his ignorance of party ideology. -By age 14 this 
had changed a lot, reasons for opinions were volun- 
teered and attempts were made to link up responses 
with earlier ones. In fact some children could give 
a running commentary when answering one of the scales. 
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7. 



Observations on Children’s Perception of political Ob - | 

jects and Problems | 



Children relate themselves in fairly definite ways to 
the political objects surrounding them, from filling them with 
distinct meanings for themselves to hardly noticing them at all. 
These patterns are listed, as they may suggest some hypotheses. 




i. Children are not aware of the community as such . The 
preliminary report stated that they cannot see social 
problems, cannot imagine how things could be different 
and do not perceive any oragnizing principle behind the 
everyday public objects they come into contact with, in 
short that everything is taken for granted. This has 
got to be qualified. For all the children see themselves 
as part of a community: their country and their town. 

In Germany the knowledge of being a German was usually 
accompanied by a lot of confused ideas about the division 
of their country (into north and south, into Berlin 
and Germany, into the DDR and a Germany that had been 
amalgamated with Belgium and Switzerland). Social prob- 
lems had entered their world as far as children were in- 
volved, e.g., starving children, children who had been 
kidnapped, and as far as they have come into direct con- 
tact with them, like a chronically sick father or an old 
age pensioner living next door. Usually though such prob- 
lems are far removed. -For the older children social prob- 
lems had made their appearance, but still only sporadi- 
cally: government should provide work for invalids, it 
should give money to the sick and provide a holiday for 
everybody in need, or stop arms purchases in favour of 
supporting the poor. 

ii . Social inequality has hardly been noticed by the young 
child . To free questions about differences between 
people - apart from the already mentioned nice-horrible 
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distinction - children think of differences between 
boys and girls, between children and adults, and only 
after after some prompting between coloured and white, 
rich and poor people. Comments about these differences 
show that at least the last one is not seen as an organi- 
zing principle of society: "poverty is only temporary", 

"it's their own fault, they must have been drinking", 
"they'd get over it, if they would only work", "the 
Queen is rich, but I am half rich". Questions about 
social classes are marred by terminological problems: 
in England class was taken to mean a school class, in 
Germany a worker's shift (Arbeiterschicht) or a layer 
of some sort. When the term was understood, the divid- 
ing lines between the child's social classes are quite 
original: "working class are those who work and non-work- 

ing class are those who are paid by the government and 
all the workers' wives", "I am lower class now, because 
I am still a child, but I'll be middle class as soon as 
I am grown up", "my Dad is industry class, I don't really 
know what my mother is, she works in a children's home". 
If a schematic drawing of a pyramid with layers labelled 
'rich' and 'poor' is used, the children found it easier 
to work with the term social class. 

To the older children social stratification had be- 
come very apparent indeed, bu.t strangely enough its exis- 
tence was denied rather often. The first response was 
usually that classes are a thing of the past, that people 
were equal today, that there were no more slaves and 
castes, and that the lower class was so small, as not to 
deserve any attention - the workers were now all in the 
middle class. Nonetheless inequality before the law was 
mentioned astonishingly frequently, (the rich being able 
to bribe the judge, etc.) and the unfair advantages the 
employers had over their workers were bemoaned. 



iii. Children are unaware of the existence of adult conflict . 
Conflict is seen only on the individual level, like boys 
fighting in the playground and adults arguing across the 
fence. They cannot conceive of clashes of interest bet- 
ween groups, such as unions and employers, religious groups 
or conflicting business groups. The world of the grown- 
ups is all harmony. Consequently they have no idea that 
much of governmental work is concerned with the regulation 
of competing interests. Children do not think of adults 
as members of possibly competing groups., everything is per- 
ceived on an individual level. 
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For instance ’strikes’: the English children had 

heard about them more vividly than the German ones, but 
these strikes weren’t a problem in their country, maybe 
in the United States. Two German boys reckoned they 
were fights among workers, with the employers figuring 
as peace makers. 

The exception to this general picture of harmony 
is war and oddly enough for the German children an up - 
rising , Wars are fought about such tangible things as 
rivers and fertile land, they are caused by quarrels 
among politicians, but they do not really happen anywhere 
near the child’s own country - this may be accounted for 
by the topicality of the Arab/Israeli war. The idea of 
an uprising and of civil disorder among the German child- 
ren appeared in various contexts: public criticism of 

the government would cause an uprising; if children were 
not loyal to their country, it could be the start to an 
uprising; Communists had something to do with uprisings; 
small religious minorities were a danger, as they might 
disturb civil harmony; if there was only one party, people 
would revolt or perish. One little girl only read the 
first page of the local newspaper to see whether there had 
been an uprising or not. -By contrast the word uprising 
was never even mentioned by the English children and 
only occurred once among the older German children. 

Perception of conflict had become more varied for the 
older children. For them conflict could be caused by 
religious differences, competition in industry, and excess 
ive complaints. Clashes of interest within one country 
that were a bit out of the ordinary were: the people 

against the government, enlisted men against the army, 
and students against the police. 



iv . Children have no distinct ideas about settling conflict . 
With conflict at the individual level, children would 
either report it to someone with more authority or they 
would simply join in and/or retaliate in the same fashion. 
With more distant incidents of conflict, like wars, their 
suggestions about solutions are limited to verbal recom- 
mendations like "they should stop", "they should make 
peace", etc. Compromises, negotiations or majority deci- 
sions to settle conflict are not very salient. Occasion- 
ally a child mentioned that discussion might help, "if 
they discuss well". No national differences on this score 
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4 years later the picture has not changed sig- 
nificantly. More emphasis is put on finding out the 
guilty party, the one who started. But the tactics 
that should be applied after that are still reliance 
on authority and force (the Bundestag should listen 
to both sides first and then intervene with force). 

v . Children have trust in the leadership of well-known 
politicians. They attribute charismatic qualities to 
some of them . The Queen for instance could and should 
make a law to stop all wars; if Adenauer, Kennedy and 
de Gaulle were still alive, there would not have been a 
war between Israel and the Ar ab s just as there were no 
wars during their life time. Or: the Prime Minister 

could satisfy everybody, if he only wanted to and if the 
country was split 50—50 on an issue, he would be clever 
enough to decide it. This trust was not found with any 
of the older children. It raises an interesting question 
in connection with the lack of criticism (see 6.V.), for 
if a child did criticize spontaneously, it was criticizing 
the Prime Minister. One possible explanation: these 

patterns are sequential. To start with trust, then the 
first attempts at criticism, then the disappearance of 
trust . 

vi. Another explanation is possible: Children make a dis- 

tinction b etween the head of state and the government , 
however vaguely . Whenever this distinction is made, 
all the trust and affection is placed in the head of 
state leaving the prime minister free to be criticized. 

If the distinction is not made, the trust is extended to 
the Prime Minister and his government. Among the German 
children the distinction was more confused and twofold 
in nature. The President and the Chancellor are seen in 
an hierarchical relation but with distinct areas of in- 
fluence. Apart from this a difference is seen between the 
state and the government. This time the state as the 
giver of pensions, sick pay etc. , is invested with all 
the affection and trust. A caution must be added: the 

term ’state 1 was misunderstood by almost half the child- 
ren, because of the German colloquialism "Vater Staat". 

It was actually seen as a person, as an organization for 
fathers and as a federation (Vereinigte Staaten) . -With 
two of the duller 14 year old children this functional 
differentiation of state and government persisted (state: 
home affairs, government: foreign affairs), to the others 

the concepts had become clear and the offices of Presi- 
dent and Chancellor were more accurately defined. 
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vii . As fits the general status of a child, he does not see 
any justification for civil disobedience of any sort . 

In the child's eyes nobody has got a right to quarrel 
with a policeman whatever the circumstances, and no child 
should disobey a teacher. To break a law is bad what- 
ever the reason and only crooks do it. - Age differences 
are pronounced in this area, as at 14 quite a few could 
conceive of situations where civil disobedience might be 
advisable. Student demonstrations, the formation of 
groups to fight specific issues, and girls who do not 
always do as told, were condoned by them. 

v i i i . Children have very little time perspective and no sense 
of history . They do not understand the concept of pro- 
gress and are quite unaware of social change. For in- 
stance, laws last about 3000 years, if made by Parlia- 
ment, and about a million years, if made by God. This 
links up with the way they take the world, as they know 
it, for granted and assume it always to have been the 
same, permanent and unmodif iab le . - Though the 14 year 
old children had acquired more historical knowledge, 
the ideas of social change and progress were still quite 
hard nuts to crack. 

ix. Children differed by nationality in the way they saw 
the relationship between government and economy . The 
English youngsters considered the determination of 
prices and intervention in the economy to be the main 
function of the government, though not quite in these 
terms. The Germans, by contrast, never thought that the 
government might have anything to do with prices. Yet 
by age 14 the German children assumed a dependent rela- 
tionship to exist between the economy and government, 
more specifically a democratic government. They all took 
it for granted that the government was in a shaky posi- 
tion, if the economy was not doing well: people would 
lose all interest in the government, poor people would 
always want more money from it without caring what form 
the government would take. The present crisis was seen 
as a test case. 

x. Children differ by age in the importance they attach to 
the image of their country . This could of course only be 
observed with the German children, as no older English 
children were interviewed. The younger German children never seemed 
to have given their country's image a thought, all the more surprising 
was the preoccupation of the older ones with this question. 






Not only did they realistically perceive the reser- 
vation in other countries toward Germany, there were 
also estimates that up to 40% of the population, 
mainly the young people, did not feel any pride in 
being German; interest in foreign affairs was justified 
with the need to make friends; one girl manifested 
what could only be called a persecution complex when 
she lamented that whatever Germany would do or did, it 
would be held against her. 

The older German children were full of democratic 
slogans . These were slogans that never turned up with 
the younger ones, like "we all are equal", "equality 
of rights in a democracy", "communists are also human", 
"one has to learn to respect other people’s opinion", 
etc. They also showed eloquent disapproval of dictator- 
ship and immediately made the association democracy- 
dictatorship. This turned out to be the influence of 
their civics course, in their view. It would have been 
interesting to see whether these slogans had been inter- 
nalized, in other words to see whether they would have 
scored as ’democrats’ on the measures proposed for area 
2 of the 'democratic values'. 
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List of Stimulus Material 



Pictorial Material: 



Children’s Playgroup with teacher. 

Nuns giving food to old people and tramps, 
UN building, UN general assembly, U Thant, 
Student sit down demonstration, Egyptians 
demonstrating against Intrepid, Soldiers 
in action, Sequence of Hitler giving a 
speech, Numerous pictures of English, 

German and internationally known poli- 
ticians, Pope, the Queen, National Symbols: 
the flag, Houses of Parliament, Nelson’s 
column, No. 10 Downing St. 



I 



Stales : 



Belief in the Bill of Rights (Remmers) , Trust in People | 

(Rosenberg), Conservatism/Radicalism (Centers), Manifest 
anti-Communism (Centers, Harvey), P.E.C. (Adorno et al ) , | 

Latent Marxism (Remmers), Political Extremism (Harvey), j 

Dogmatism (Rokeach) , F-Scale (Adorno et al.), R.-T. factors { 

(Eyseck), Patroitism (Harvey), Anomy, (Srole) , Sense of j 

Political Efficacy, Sense of Citizen Duty, Subjective j 

Competence (Almond /Verby) , Confidence (Oppenheim) , I 

Parental Behaviour (Bronf enbrenper) , Rigidity (Leach), j 

Political Apathy (Rosenberg, Harvey). | 




I i 



Selected questions from: 

The Development of basic attitudes and values toward 
Government and Citizenship during the Elementary school 
years, by R.D. Hess, J.V. Torney; "Zur Wirksamkeit 
politischer Bildung", Institut fur sozialf or schung an der 
Goethe Universitat, Frankfurt; 

Hochschule fur Internationale Padagogische Forschung, 
Fragebogen fur Primaner; 



\hr ied : Tendency to generalize (Leach), 

Rule-breaking Test (Leach), 

Moral Judgement Test (Leach) , 

Semantic Differentials of Politicians, and Teachers. 
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SPECIFIC SUBJECT MATTER HYPOTHESES AND QUESTIONS 
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APPENDIX I 



SPECIFIC SUBJECT MATTER HYPOTHESES AND QUESTIONS 



This is a list of the hypotheses and questions suggested by 
the subject matter committees. 



i. Science 



1. To what extent are the newer science teaching methods 
based upon investigation and student enquiry achieving better 
results than traditional methods? 



2. Are any differences observed more marked in achieving 
some objectives than others? 



3. To what extent is the success of science teaching, and 
particularly the newer methods of science teaching, dependent upon 



a. the material conditions under which science is taught, 
e.g. provision of laboratories and apparatus 

b. the time allocated to science and the fraction of it 
spent in practical work in the school timetable, 

c. the provision of laboratory assistance, 



d. the training teachers have received and the facilities 
for refresher courses? 



4. Is a more adventurous attitude towards future careers 
in science, as indicated by a wider range of desired careers, 
favoured by the newer teaching methods? 



5. What differences can be detected between the sexes in 
their attitudes to, and their achievements in: 



a. 

b. 



science as a whole, 

the physical sciences and the life sciences. 



6. To what extent are any of these differences related to: 



a. 

b . 

c . 

d. 



social variables, 

home background, including conditions and interests, 
school administration and organization 
teacher variables? 
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7. To what extent is achievement in, and attitudes towards, 
science related to: 

a. the level and rate of advance of technological develop- 
ment in the countries concerned. 

b. rural and urban environments? 

8. To what extent is achievement in, and attitudes towards, 
science related to school organization variables such as comprehen- 
sive and selective systems of education? 

9. The highest levels of science achievement will be attained 
by a small percentage of full-time students in countries in which 

a smaller proportion of the relevant age group is still attending 
school. 



10. Where the lower mental process score is held constant, 
performance on the higher mental process score will be directly 
related to: 



a. the quantity and quality of the training of the teachers. 

11. Where the knowledge and skill achievement in science 
is held constant, performance on the more complex objectives will 
be directly related to: 

b. teachers ratings of the student's opportunity to learn 
items involving more complex objectives. 

12. Achievement on basic science skills and basic content 
is unrelated to socio-economic status of students (while achieve- 
ment on the more complex objectives is directly related to 
socio-economic status). 

13. The role of women in a society as indicated by 
occupational freedom and educational opportunities will determine 
the differences in the interests, attitudes and achievements in 
science of boys vs. girls. 

a. In all countries, there will be no differences in 
overall science achievement of 13-year-old boys and girls 
but there will be slight differences favouring the girls 
on highly verbal problems and slight differences favouring 
the boys on computational problems. (Hold constant the 
level of science instruction). 
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b. the following will vary from no difference to marked 
difference depending on the role differentiation of women 
in the countries . 

girls will differ from boys in interest in science; 
girls will differ from boys in plans to take further 

scienc.e; 

girls will differ from boys in attitudes toward science; 
girls will differ from boys in view about who can learn 

science; 

girls will differ from boys in science achievement; 
girls will differ from boys in the higher and lower 

mental process scores. 

The above differences between boys and girls will be 
greatest when the students are in single-sex schools; the differences 
wil be least when they are in co-educational schools. 

15. The above differences between boys and girls will be 
greatest for students whose parents have completed only elementary 
education while the differences will be least for students whose 
parents have completed secondary or more education. 

16. Indices of the nations* urbanization, geographical and 
occupational mobility, and technical-industrial development of 
the countries, will be related to: 

a. science achievement at terminal stage, but not at 
14-year-old level. 






1 . 

• • 
ii. 



in. 

iv. 

v. 

vi . 



Ui. 



b. interest in science and desire to take more science 

c. attitudes toward the role of science in the society 

d. description of science teaching and learning 

e. view of science as a process 

f. views on who learns science 

g. attitude toward school learning 

h. performance on higher mental processes, but not on 
lower processes. 

17. Students who plan (or desire) to enter scientific and 
technical occupations will differ from students who plan (or 
desire) to enter humanistic and non-technical occupations in: 
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a. interest in science 

b. wish to take more science 

c. attitudes toward the place of science in society 

d. attitudes toward a science as a process 

e. attitudes about the difficulties of learning science 

f. total science score 

g. higher mental processes 

h. lower mental processes. 



ii. Reading Comprehension 

1. Internal structure of the test battery: 

a. What are the relationships between the different tests 
of reading comprehension, in the different countries and how 
do these relationships change according to the level of educa- 
tion of the target populations? 

b. What are the relationships between reading comprehension, 
rate of reading, vocabulary, adjustment of rate of reading, in 
the different countries; how do they change according to the 
level of education of the target populations? 

2. Educational practices and background: 

The level of reading comprehension may be influenced 

by: 

a. methods of instruction at the beginning of reading 
(i.e. analytic, synthetic, mixed) 

b. pre-school instruction in reading 

c. importance of reading in the curriculum (definite 
reading lessons) 

d. formal training in grammar, in vocabulary 

e. kinds of reading assignments if any (literary apprecia- 
tion, extensile reading, memorizing poems or texts, etc.) 



f. number of teachers the child has each week. 

g. type and size of school 

h. type of studies (i.e., general, technical ) 
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i. specific subjects studied 

j . grade 

k. advance or retardation in studies 

l. the grade at which a second or foreign language is 
introduced . 



3. It may also be influenced by the availability of 
reading material: 

a. in the school (does the child have text books for 
different subjects , is there a school library?) 

b. in the home (books, newspapers, magazines) 

c. in the community (can books, newspapers, magazines 
be bought or borrowed?) 

4. The environment outside the school may have an in~ 
fluence on reading comprehension: 

a. Occupation and social standing of the parent (s) 

b. Level of instruction of the parent (s) 

c. Amount of exposure to mass media (radio, TV, films) 

d. Is the language of instruction the child's first 
language? 

e. Is the child monolingual? 



5, The interests of the child and his attitudes may 
have an influence on reading comprehension: 

a. Preference for some of the subject matters (by direct 
ranking, by ranking of the different texts of the test, etc.) 

bo Interests in leisure activities 

c. Attitudes toward reading (child's idea of the aims 
of reading, why it is important to be able to read, etc.) 

do Ambitions and aspirations of the child 

e. Attitudes of the child toward school (does he regard 
it as a compulsion) 

6. Study and reading habits of the child may influence his 
reading comprehension (e.g. finger tracing, vestigial oral reading, 

etc) 
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7. There probably is a relationship between the 
child's achievement in different subjects and/or overal— 
achievement and his level of reading comprehension. 

8. Physical defects might impair the child's read- 
ing. (e.g. eye sight, hearing, speech defects) 

iii . French as a foreign language 

1. (i) Variation in achievement profile will reflect 

variation in teaching methods and objectives: children 

taught French by audio-visual methods will reach a higher 
level of achievement in speaking and listening skills; 
children taught French by gr ammar- tr ansla t ion methods will 
reach a higher level of achievement in reading and writing 
skills . 



(ii) This variation will not persist into the pre- 
university grade (if length of exposure to French is held 
constant ) 

2. Level of achievement in French will be related to 
length of exposure to French rather than to chronological 
age or grade level. 

3 „ Attitudinal factors (in both home and school en- 
vironment) will affect level of achievement in French. 

4. Socio-economic factors (parental education, socio- 
economic skills, etc.) will be related to child's level of 
achievement in French. 

5. Girls will reach a higher level of achievement in 

French than will boys: this difference will be accentuated 

in single-sex schools. 

6. Size and composition of French class will be re- 
lated to achievement in French. 

7. Selection procedures will be related to level of 

achievement in French: children in streamed schools will 

achieve a higher mean score than will those in comprehen- 
sive systems; the latter will produce a higher total 
"yield" but a lower average score. 

8. Children's level of achievement in French will 
related to the teacher's training, qualifications, and 
teaching experience. 
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9. Variation in economic support for education will 
be related to variation in level of achievement in renc 

10. Children’s level of achievement in French will be 
related to parental aspirations and attitudes. 

11. Level of achievement in French will be related to 
opportunities for contact with French speaking groups. 

12. Level of achievement in French will be related to 
the "linguistic distance" between French and the mother 

tongue . 

13. Measures of verbal ability in the child's mother 
tongue will correlate highly with measures of achievement 
in French at the advanced levels; in the early stages of 
learning, there will be a much lower correlation. 



iv . 



English as a foreign language 



1 The greater the amount of assistance provided 
the country in teaching English (in terms of guidance, 
supervision, facilities and equipment), the higher the 
cational achievement in English will be. 
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2. The degree of linguistic complexity of the country 
(number of languages or distinct dialects spoken) will . 
affect the level of achievement in English, especially in 
the early stage of learning English. 



3. The linguistic distance between the country s 
(native) language and English (as established by expert 
linguists) will be negatively correlated with the level o 
general achievement in English. 



4. The country's need for English as a medium of 
communication in international trade (including tourism), 
science and technology will be positively correlated to 
(i) the achievement in English and (ii) the amount o 
assistance provided by the country - see also Hyp. 



5. The quality of teacher training will be posi- 
tively correlated with the achievement in English. 

6. Schools having special English teachers will per 
form better than schools having teachers teaching other 
subjects besides English. 
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7 . The attitude of the teacher will be positively 
correlated with the students' achievement in English. 

8. The effect of size of class on the achievement 
in English will vary with the different sub-tests. 

9. The method of teaching will affect the level of 
achievement in English. 

10. The linguistic background of the student will 
affect his level of achievement in English especially in 
the early stage. 

11. Age at which the, student starts English will be 
negatively correlated with his achievement in English - 
holding constant the number of years of English training, 
those students who started at a younger age will do better 
than those who started later. 

12. There will be an optimum range in the degree of 
concentration in learning English (or intensity in terms 
of the number of hours of English classes per year), and 
the relationship between degree of concentration and level 
of achievement will be curvilinear. 

13. There will be a positive correlation between the 
socio-economic status of the parent and the student's 
achievement in English. The correlation, however, will be 
higher than that between socio-economic status and achieve- 
ment in mathematics. 

14. The general ability of the student, in terms of 
(i) grade point average, and (ii) I.Q. will be positively 
correlated with his achievement in English. The corre- 
lation, however, will be lower than that between general 
ability and achievement in mathematics. 

15. The attitude of the student toward the English 
language and toward the culture associated with the lan- 
guage will be positively correlated with his level of 
achievement in English. 

16. Attainment in particular aspects of English - 
reading, writing, understanding and speaking - will vary 
directly with the kind of method employed in teaching — 
grammar / translation , or audio-visual. (c.f. Hyp. 1) 

17. Attainment among individual students will be posi- 
tively related to the degree of specialized selection em- 
ployed in forming a class. 
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v , Literature 

1. Characteristic patterns of expressed response 
will be closely related to knowledge about and attitudes 
toward literature. 

2. The various outcomes of literary education are 
related to, but not identical with each other. 

3. (i) International differences in the stated aims 
of literary education are related to the place of litera- 
ture in a nation’s culture and practices of literary edu- 
cation. 

(ii) International differences in the outcomes of 
literary education are more closely related to the practices 
of literary education than to the official statement of 
its aims. 



4. Within and between nations, outcomes of literary 
education will vary in relation to social, cultural and 
individual characteristics of students. 

5. Internationally, the differences between elite and 
popular culture as represf ited in the schools is related 

to the place of each culture in the society as a whole. 



vi . Civic Education 

1. Student knowledge of governmental affairs is high- 
er when civic education is given as a separate subject than 
when it is an overall aspect of a number of subjects. 



( , 2. The students' interest in participating in social 

j activities is positively related to their interest in 

; governmental affairs. 

| 3. Students' participation in extra-school activities 

( (clubs, games, etc.) is related to their perception of 

| authority relationships within the family. 

4. The more a curriculum in civic education stresses 
a functional approach (as opposed to a structural approach) 
1 , the greater will be the variability of the knowledge-pro- 

file pattern. 




fr 
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5. The variability of knowledge of international rela- 
tions is positively related to the degree a curriculum in 
civic education stresses a structural approach 
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6. The student’s interest in the subject civics 
is higher when instruction is given in short, concen- 
trated courses, than when instruction is spread out over 
a whole year (or more) . 

7. Student differences between age 14 (+1) and age 

17 (+1) in attitudes towards participation in social acti- 

vities is positively related to the type of school organi- 
zation which promotes such activities. 

8. There is a negative correlation between the stu- 
dents’ knowledge of international organizations and their 
thinking in stereotyped terms. 
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THE ANALYSIS AND PRESENTATION OF THE IEA EVIDENCE 
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Our final object is to write a clear and agreed report in 
which our evidence is presented and analysed, so that we can draw 
some conclusions and raise some questions-. This paper considers 
some of the problems that we must overcome on the way to our goal. 

I. DESCRIPTION 



1. How Should we Present the Weights., 

Our evidence will be derived from samples, but our interest 
lies in the populations that these samples represent. Our samples 
should be probability samples, which means that every pupil, teacher 
and school in a population should have a chance, which we can 
specify, of appearing in the sample. The chances are specxxxeu by 
the overall sampling fractions, and the reciprocals of the sampling 
fractions, which are called the raising factors, tell us how many 
pupils/teachers/schools in the population are represented by a pupil/ 
teacher/school in the sample. In the analyses either the raising 
factors themselves, or any set of numbers proportional to them, can 
be used as the weights. The constant of proportionality makes no 
difference. But for the presentation we must decide what cons tan 
would be least likely to mislead the reader. Among the possible 
choices two stand out. Thus in the case of pupils we could either 
choose the constant so that the sum of the weights was equal to the 
actual number of pupils in the sample, or we could choose it so that 
the sum of the weights was equal to the number of pupils in the 
simple equivalent sample. The ratio of these two is the design 
effect, and its square root is the number by which simple random 
sampling estimates of the standard errors should be multiplied to 
give the estimates of the actual complex standaxd errors. If it 
could be assumed that the design effect were well understood there 
would be no problem. But this is not yet the case, as was clear from 
some of the comments on our first report, and on the whole the safer 
course seems to be to make the weighted sum equal t0 t he actual sum ’ 
and to print this together with the design effect. The decision on 
this point should be taken before the evidence is put into the computer, 
although the design effects will only be known after a certain amount 
of analysis has been done. 




2. The Component Analysis. 

At an early stage the component analysis between the country 
means of the part scores be carried out, to see whether the part scores 
need to be re-weighted. If the component weights do not differ 
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appreciably from the assigned weights no re-weighting will be needed. ! 

But if re-weighting is required it should clearly be done at the 
earliest possible stage. 
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3. Analyses Over All and Between and Within Schools, 

Within each country we can make three kinds of analysis. In 
the first place there is the kind which for want of a better name is 
called over all. In this the measures for each pupil are used, dis- 
regarding (except in the calculation of the design effects) the fact 
that each pupil is a member of some school. Secondly we have 
analyses between schools; for which the measures for pupils in the 
same school are aggregated, so that differences within the same 
school disappear from the analysis. Finally there are analyses within 
schools, for which the school mean is subtracted from each pupil 
measure, so that differences between schools disappear. Analyses 
between and within schools are complementary to the analyses over all. 
The first stresses the differences between schools and their neighbor- 
hoods , by eliminating the differences between pupils in the same school. 
The second stresses the differences between pupils in the same school, 
by eliminating the differences between schools. 



4. Analyses Between Countries . 

In principle we could also put together the evidence from 
different countries by analyses over all, between countries and within 
countries. Whether this would be useful depends upon the relative 
extent of the common features and the differences between countries. 

If the differences greatly preponderated a more piecemeal procedure 
might be preferable. In any case it seems clear that in putting to- 
gether the evidence from different countries each country should be 
given the same weight. There is little to be said for making the 
country weights proportional to population, and nothing to be said 
for making them proportional to the size of the sample. On the other 

hand it seems better to add covariance rather than correlation matrices. f 

| 

1 



5. How much Detail? 

How much detail should be printed in the computer output? There 
are two aspects to this question. On the one hand it is plainly 
necessary that members should have the fullest possible access to the 
evidence, so that the agreed report can be based on the combined wisdom 
of the whole body. On the other hand the report itself must not be 
swamped by detail. This means that the output for circulation to 
members should be more extensive than the output included in the final 
report. But how much more extensive? 
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Univariate Statistics. 



The first reauisite will be the information about the measures 
separately - the univariate statistics. A member will 
the name of each variable, and its c 1 ?’ P P It ould be con - 

the scaled variables in a rirs ( f 0 f DU pil) . 

rr i B ^n -fc; indicated doubt may arise (e.g., tor sex or pupx ; 

^eTean and ^ “on could 

distributioi^and SSSS^S. For Scaled variables the second 
lift could give the proportions in the previous ^categories A, *,J, 
etc. , while the standard deviations ^thefirstli lnterpret - 

the dichotomies A versus not A, etc. These are neea a y 

ation if the regression coefficients are in standard form. Thus i 
bovs count high in the variable Sex of pupil a regression coefficient 
of 0.1 means that boys score 0.* more than girls, where P is the 
proportion ofboysandthe scores are in^ard^^. above that of 
general unsealed case tms applies . . . ^ a , rp the 

the class taken as s^ndard so that d ‘ c ^ t ^“ arlable has on ly 

two" values ^bu t ^ in S standard form they are not (0,1) but (0,1/Vpq)- 



6.1 These are the univariate statistics for the analyses 
over all, and those for the analyses between schools and within 
schools could be displayed in the same way. Alternatively, and P 
. r pf erablv they could be indicated by merely adding, to the 

tab le^of ^student variables, a column giving the proportion^f the 
variance that lay between schools. For the s ^o^/^^ v ^i S ables> 
nronortion is of 1 course unity, and so it is for the teacner vc i 
since these are only to be considered when averaged over schoo s. 



7 . Bivariate Statistics. 



The next step is to obtain the bivariate statistics, or in other 
words Ihe correlation matrices. These will be in three forms, namely, 

overall? between schools and within schools ^X^ftch o!s, 

Qrr « rQ Tn the over all case the sraDixiuy ui .. . . 

obtained by replication, in the way indicated on page o u 

No. XI. 
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/•I These correlation matrices will contain a great deal of 
the information about the relations between the variables, and will 
be the foundation for the subsequent regression analyses. A 
scrutiny will show which are the largest correlations and where these 
large correlations are of particular interest they could be displayed 
either as scatter diagrams or in tabular form. Display in these 
forms is illuminative and suggestive. On the other hand it needs so 
much space that it can only be applied to a fairly small number of 
carefully chosen relations. 



| 8. Interaction. 

I 

[ The next step is to decide whether any, and if so what, product 

I terms should be included in the regressions. The point here is that 

J if there is an interaction between two variables it can only emerge 

| from the regression if the corresponding product term is included. 

| We know from past experience that most interactions are quite small, 

| particularly when unsealed variables are used in cases where the 

| assumption of linearity seems likely to be false. We cannot put all 

| the product terms into the regression equation, because there are far 

too many of them; twenty variables give 190 product terms. 

| Fortunately there has recently been developed a complementary mode of 

| analysis whose chief merit is that it can reveal large interactions 

I when they exist. This method proceeds by making successive splits 

I - of the total sample, which is finally resolved into a number of 

| clusters such that each cluster is defined by ranges of values of 

| certain of the predictor variables and contains pupils with a much 

| smaller variation on the criterion than is found in the whole sample, 

j The process begins by finding that predictor, and the split on that 

| predictor, which divides the sample into two clusters such that the 

f criterion variation between them is a maximum, and consequently the 

| variation within them a minimum. It then takes one of these clusters 

j and splits it again in the same way, and so on until an unsplitable 

j cluster is reached - a twig, so to speak, at the end of a branch - 

after which it returns to reduce the other branches to twigs. A 
! cluster becomes a twig when it is either too small or too homogeneous 

j to be split further, or when none of the predictors will split it. 

In principle this process of resolving the whole sample into a set 
of most homogeneous clusters is much simpler than multiple regression, 
I and it does not impose the same restrictions or assumptions on the 

s variables. But it is also plain that in practice it is extremely 

! complicated if there are more than two or three variables, and it has 

only recently been computerized through the work of Sonquist and 
| Morgan at the Survey Research Center at Michigan. In their monograph 

(No. 35) these authors describe the program and also give the results 
when it was applied to nine survey examples previously analyzed in 
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other ways, to illustrate its working. These examples show that while 
the analysis by successive splits is generally in broad agreement with 
the analysis by regression it does sometimes reveal the existence of 
large interactions which quite alter the interpretation. The test for 
the existence of interaction is lack in symmetry in the tree. If 
there is little interaction the tree will be symmetrical. If there 
is much it will be lopsided, with branches of different lengths. An 
attractive feature of the output is that each final group can be 
described completely, in terms of the predictor variables, by follow- 
ing the branch that leads to it. It may not be unduly sanguine to 
think that this may give us some new insights, apart from the question 
of discovering interactions. 

8.1 This form of analysis is described at somewhat greater 
length in IEA/B/40 with a discussion of some of the problems of apply- 
ing it. We should gain some extremely useful experience with it on the 
dry run, and it will also be applied to the second Plowden survey 
before the main I.E.A. survey takes place. 



9 . The Regression Analyses. 

After the analysis by successive splits has shown what inter- 
action terms, if any, are needed the regression analyses can be 
carried out. These will give the alternative description of the 
evidence. Putting the two descriptions side by side will be like 
looking at the photographs of the same landscape in different lights. 
Each may reveal features hidden in the other . 

9.1 So far we have been concerned with the description of 
our evidence, in terms of the relations between the variables. These 
relations are exceedingly complicated, and economical and compact 
descriptions of them are needed before we can undertake the second 
and more difficult part of our task, which is to infer possible chains 
of causation. In this part of the task we must be guided not only by 
the immediate evidence which we have summarized, but also by our 
preconceptions, which in turn are based on what we know of the 
previous evidence and also on a very wide range of partly remembered 
experience. Our object is to see to what extent the immediate 
evidence should modify our preconceptions, or, in other words, to test 
our hypotheses. 
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XI. INFERENCE 



10. Prima facie and Background Evidence. 

When we turn from description to inference what can be the 
general line of our argument? Part of our evidence will be con- 
cerned with variations in school practices, such as methods of 
teaching, organization, and the training of teachers. A second part 
will be concerned with school outcomes, such as the pupils achieve- 
ments and attitudes. A third part will be concerned with the back- 
grounds against which the practices operate to produce the outcomes. 

We have chosen our variables to represent these practices, outcomes 
and backgrounds because our general experience leads us to believe 
that some practices may produce better outcomes than others. These 
beliefs are embodied in our hypotheses, and our object is to see 
whether our evidence supports the hypotheses, and, if so, how strongly. 
We can begin, in any instance, by seeing whether there is a prima facie 
case. This will be so if there is a positive association between the 
outcome variable and the school practice variable. If there is no. 
such association, or if the association is negative, the prima facie 
evidence tends to confute our hypothesis and we must ask whether 
the hypothesis is mistaken or whether the evidence in its favor is 
masked by some other factors. We should not lightly abandom beliefs 
to which our general experience has led us . But equally fairness 
demands that we should scrutinize the instances where the prima facie 
evidence is favorable to our beliefs, to make sure that it is not 
delusive. In both kinds of instance the consideration of the prima 
facie evidence is only the first step. We must go on to consider the 
background evidence. Thus if we are concerned with boys and girls 
aged 14 we must remember that much has already happened to these 
children before they come within our view. For example they have 
been conceived, born, and brought up by their parents in certain 
neighborhoods, and they have undergone primary and two or three years 
of secondary schooling. In any particular instance all, or some, of 
these events may be associated with the school practice, or the out- 
come, or with both, and taking account of them may change the strength, 
or even the direction of the prima facie evidence. 



I ll . Surrogates for Past Events. 

Ideally we ought to take account of all these events, but 
I plainly we can only°do so to the extent that they are represented in 

I our evidence. Generally speaking they are not directly represented, 

I but some of them are represented indirectly by surrogates, such as the 

| present parental occupation and the kind of neighborhood in which the 

I 
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family now lives. Others raise difficult questions. For example it 
is now firm evidence that at conception a child receives a certain 
genetic endowment, which may be one of the most important factors 
governing his future life. Xs it fair to say that in a country with 
a selective system of secondary education the type of secondary school 
is to some extent a surrogate for genetic endowment, as well as for 
primary schooling? And in a country where secondary education is not 
selective are there other variables, such as the pupils stream or 
track or type of course that can play the same part? Or is this 
argument circular? To what extent is it legitimate to use one test 
as a surrogate for background in evaluating outcome measured by another 
test, on the grounds that the first test represents mainly the effects 
of background while the second represents mainly the effects of 
schooling? If we do not use surrogates for background we run the risk 
of attributing the schooling outcomes that are the effect of background. 
But if we do use surrogates they are likely to be either too weak, 
because they fail to cover much of the background, or too strong 
because they include some foreground as well. Perhaps we should use 
both weak and strong surrogates, to produce upper and lower estimates 
of the effects of schooling. Thus a possible surrogate for background, 
particularly for the foreign language test, is the reading test in the 
native language, which certainly represents background to a large 
extent, But it may be too strong because it also includes a good deal 
of foreground. 



12. Cause or Effect? 



Another difficulty arises about relations of outcomes of 
different kinds. Do attitudes determine achievement, or does achieve- 
ment determine attitudes? Or is the relation reciprocal? These are 
difficult questions, which need the combined wisdom of our members, both 
in the choice of variables before the analyses are carried out and 
afterwards in the interpretation of the results. 



13 . Regression as Analysis of Variance. 

Let us now turn to the simpler question of how the analyses 
should be carried out when the choice of variables has been made. Our 
general principle will be to analyze the variation of the outcome, to 
see how much of it we can associate with the school practice variables, 
how much with the background variables, and how much we cannot account 
for because it is associated with events for which we have neither direct 
measurements nor even surrogates, or which the variation associated 
jointly with the predictors is divided into successive increments 
associated with them individually, one must produce arguments not only 
to justify the choice of predictors but also to justify the order in 
which they are taken. As a rule, though not invariably, the earlier 
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